search

projectkudu / kudu

Kudu is the engine behind git/hg deployments, WebJobs, and various other features in Azure Web Sites. It can also run outside of Azure.
Apache License 2.0
3.12k stars 653 forks source link

Failed to git push to App Service on Linux #2251

Closed shibayan closed 4 years ago

shibayan commented 7 years ago

If the repository size if large git push will fail.

When pushing about 10MB repository, the following trace was output.

/home/LogFiles/kudu/trace>cat 2016-12-06T05-15-32_8e30c7_024_POST_shibayan-linux.git-git-receive-pack_200_4s.xml
<step title="Incoming Request" date="2016-12-06T05:15:32.071" instance="8e30c7" url="/shibayan-linux.git/git-receive-pack" method="POST" type="request" pid="11,1,56" Connection="close" Transfer-Encoding="shibayan-linux.scm.azurewebsites.net" Content-Type="application/x-git-receive-pack-request" Accept="application/x-git-receive-pack-result" Accept-Encoding="gzip" Host="shibayan-linux.scm.azurewebsites.net" User-Agent="git/2.10.2.windows.1" X-SITE-DEPLOYMENT-ID="shibayan-linux" WAS-DEFAULT-HOSTNAME="shibayan-linux.scm.azurewebsites.net" X-Client-IP="131.213.200.176" X-Forwarded-Host="shibayan-linux.scm.azurewebsites.net" X-Forwarded-Server="SmallDedicatedLinuxWebWorkerRoleIN253.corp.microsoft.com" >
  <step title="RpcService.ReceivePack" date="2016-12-06T05:15:32.318" >
    <step title="Executing external process" date="2016-12-06T05:15:32.401" type="process" path="git" arguments="rev-parse --git-dir" /><!-- duration: 945ms -->
    <step title="Assuming git repository at /home/site/repository" date="2016-12-06T05:15:33.360" /><!-- duration: 21ms -->
    <step title="Executing external process" date="2016-12-06T05:15:33.537" type="process" path="git" arguments="rev-parse --git-dir" /><!-- duration: 58ms -->
    <step title="Assuming git repository at /home/site/repository" date="2016-12-06T05:15:33.608" /><!-- duration: 29ms -->
    <step title="Creating temporary deployment" date="2016-12-06T05:15:34.420" /><!-- duration: 188ms -->
    <step title="GitExeServer.Receive" date="2016-12-06T05:15:34.621" >
      <step title="Executing external process" date="2016-12-06T05:15:34.637" type="process" path="git" arguments="receive-pack --stateless-rpc &quot;/home/site/repository&quot;" >
        <step title="Process dump" date="2016-12-06T05:15:35.346" exitCode="128" type="processOutput" /><!-- duration: 17ms -->
      </step><!-- duration: 739ms -->
    </step><!-- duration: 771ms -->
  </step><!-- duration: 3164ms -->
  <step title="Error occurred" date="2016-12-06T05:15:35.498" type="error" text="fatal: The remote end hung up unexpectedly

/usr/bin/git receive-pack --stateless-rpc &quot;/home/site/repository&quot;" stackTrace="  at Kudu.Core.Infrastructure.Executable.Execute (Kudu.Contracts.Tracing.ITracer tracer, System.IO.Stream input, System.IO.Stream output, System.String arguments, System.Object[] args) [0x000a0] in &lt;ed7b12c633004783b89817b8de914f0c&gt;:0 
  at Kudu.Core.SourceControl.Git.GitExeServer.ServiceRpc (Kudu.Contracts.Tracing.ITracer tracer, System.String serviceName, System.IO.Stream input, System.IO.Stream output) [0x00026] in &lt;ed7b12c633004783b89817b8de914f0c&gt;:0 
  at Kudu.Core.SourceControl.Git.GitExeServer.Receive (System.IO.Stream inputStream, System.IO.Stream outputStream) [0x00018] in &lt;ed7b12c633004783b89817b8de914f0c&gt;:0 
  at Kudu.Services.GitServer.ReceivePackHandler+&lt;&gt;c__DisplayClass3_0.&lt;ProcessRequestBase&gt;b__0 () [0x000de] in &lt;251a6e97992d4151b50db7cd589803a2&gt;:0 
  at Kudu.Contracts.Infrastructure.LockExtensions.TryLockOperation (Kudu.Contracts.Infrastructure.IOperationLock lockObj, System.Action operation, System.String operationName, System.TimeSpan timeout) [0x00033] in &lt;61029332118641a5936bcf8f96b27004&gt;:0 
  at Kudu.Contracts.Infrastructure.LockExtensions.LockOperation (Kudu.Contracts.Infrastructure.IOperationLock lockObj, System.Action operation, System.String operationName, System.TimeSpan timeout) [0x00000] in &lt;61029332118641a5936bcf8f96b27004&gt;:0 
  at Kudu.Services.GitServer.ReceivePackHandler.ProcessRequestBase (System.Web.HttpContextBase context) [0x00078] in &lt;251a6e97992d4151b50db7cd589803a2&gt;:0 
  at Kudu.Services.GitServer.GitServerHttpHandler.ProcessRequest (System.Web.HttpContext context) [0x00007] in &lt;251a6e97992d4151b50db7cd589803a2&gt;:0 
  at System.Web.HttpApplication+&lt;Pipeline&gt;c__Iterator1.MoveNext () [0x00dd7] in &lt;d3ba84a338d241e2ab5397407351c9cd&gt;:0 
  at System.Web.HttpApplication.Tick () [0x00000] in &lt;d3ba84a338d241e2ab5397407351c9cd&gt;:0 " /><!-- duration: 17ms -->
  <step title="Outgoing response" date="2016-12-06T05:15:35.603" type="response" statusCode="200" statusText="OK" /><!-- duration: 15ms -->
</step><!-- duration: 3561ms -->

Looking at the trace log, an error occurred when receive-pack was called twice. Increasing the size of http.postBuffer with git config avoided errors.

git config http.postBuffer 52428000

For reference, it is an error log that emphasizing yellow. You can see that receive-pack has been called multiple times. (This is the normal behavior of git)

kudu trace logs

davidebbo commented 7 years ago

Yes, there is indeed some issue with large repos on Linux. @naziml may have more to say about it, though I don't think it's been solved.

Related thread: https://social.msdn.microsoft.com/Forums/en-US/66d136c4-34c7-43c4-b2f8-7ddd842a9996/cannot-push-largeish-git-repo-to-app-service-linux?forum=windowsazurewebsitespreview

shibayan commented 7 years ago

Thanks. If the image is included, the size of the repository will increase, so I think it is a problem to be fixed. If I can install custom Kudu, I may be able to help with this problem :)

I'm very excited about App Service on Linux and Docker support.

davidebbo commented 7 years ago

/cc @nickwalkmsft who will help out with Kudu/Linux is the coming weeks.

snobu commented 7 years ago

Interesting finding. My 32MB repo deploys just fine and i did it quite a few times.

tmp$ du -sh gifinator
32M     gifinator

gifinator$ git count-objects -vH
count: 0
size: 0 bytes
in-pack: 254
packs: 1
size-pack: 24.32 MiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

Could be more to this story than just the repo size?

shibayan commented 7 years ago

I was interested in this problem and I was investigating the cause.

This may be a problem with Mono / XSP. It is very troubling. When receiving a request with Transfer-Encoding: chunked, Mono seems to always return empty as InputStream.

However, it seems that this problem can be avoided by combining nginx and fastcgi-mono.

So it is a bug in mod_mono.

shibayan commented 7 years ago

I investigated further, but this is a problem on Mono side. It seems only to wait for referencesource to be acquired or to avoid it with FastCGI.