[Windows] Deploying with submodules: "forked process <number> died unexpectedly"

[ashmind]

Repro steps.

Pushed update into my GitHub repo, Kudu source refresh failed on git submodule sync (before any other build steps). Same command fails when run through SCM.

Same command works OK on local.

Azure GIT: 2.14.1.windows.1 Local GIT: 2.8.1.windows.1

Project structures.

https://github.com/ashmind/SharpLab/tree/edge

The log/error given by the failure.

Deployment Log: Updating submodules

 1 [main] sh 14460 fork: child -1 - forked process 19080 died unexpectedly, retry 0, exit code 0xC0000142, errno 11\nD:\Program Files (x86)\Git\mingw32/libexec/git-core\git-submodule: fork: retry: Resource temporarily unavailable\n1062136 [main] sh 14460 fork: child -1 - forked process 14188 died unexpectedly, retry 0, exit code 0xC0000142, errno 11\nD:\Program Files (x86)\Git\mingw32/libexec/git-core\git-submodule: fork: retry: Resource temporarily unavailable\n      2 [main] sh 12636 fork: child -1 - forked process 15960 died unexpectedly, retry 0, exit code 0xC0000142, errno 11\nD:\Program Files (x86)\Git\mingw32/libexec/git-core\git-submodule: fork: retry: Resource temporarily unavailable\n1077157 [main] sh 12636 fork: child -1 - forked process 17448 died unexpectedly, retry 0, exit code 0xC0000142, errno 11\nD:\Program Files (x86)\Git\mingw32/libexec/git-core\git-submodule: fork: retry: Resource temporarily unavailable\n3129316 [main] sh 12636 fork: child -1 - forked process 14584 died unexpectedly, retry 0, exit code 0xC0000142, errno 11\nD:\Program Files (x86)\Git\mingw32/libexec/git-core\git-submodule: fork: retry: Resource temporarily unavailable\n7170343 [main] sh 12636 fork: child -1 - forked process 16196 died unexpectedly, retry 0, exit code 0xC0000142, errno 11\nD:\Program Files (x86)\Git\mingw32/libexec/git-core\git-submodule: fork: retry: Resource temporarily unavailable\n15231702 [main] sh 12636 fork: child -1 - forked process 10816 died unexpectedly, retry 0, exit code 0xC0000142, errno 11\nD:\Program Files (x86)\Git\mingw32/libexec/git-core\git-submodule: fork: Resource temporarily unavailable\n18500513 [main] sh 14460 fork: child -1 - forked process 7284 died unexpectedly, retry 0, exit code 0xC0000142, errno 11\n/mingw32/libexec/git-core/git-sh-setup: fork: retry: Resource temporarily unavailable\n19559510 [main] sh 14460 fork: child -1 - forked process 10388 died unexpectedly, retry 0, exit code 0xC0000142, errno 11\n/mingw32/libexec/git-core/git-sh-setup: fork: retry: Resource temporarily unavailable\n21621214 [main] sh 14460 fork: child -1 - forked process 20216 died unexpectedly, retry 0, exit code 0xC0000142, errno 11\n/mingw32/libexec/git-core/git-sh-setup: fork: retry: Resource temporarily unavailable\n25667323 [main] sh 14460 fork: child -1 - forked process 2244 died unexpectedly, retry 0, exit code 0xC0000142, errno 11\n/mingw32/libexec/git-core/git-sh-setup: fork: retry: Resource temporarily unavailable\n33729144 [main] sh 14460 fork: child -1 - forked process 15828 died unexpectedly, retry 0, exit code 0xC0000142, errno 11\n/mingw32/libexec/git-core/git-sh-setup: fork: Resource temporarily unavailable\n\r\nD:\Program Files (x86)\Git\cmd\git.exe submodule sync

SCM (Manual):

D:\home\site\repository>git submodule sync
      1 [main] sh 3448 fork: child -1 - forked process 1116 died unexpectedly, retry 0, exit code 0xC0000142, errno 11
D:\Program Files (x86)\Git\mingw32/libexec/git-core\git-submodule: fork: retry: Resource temporarily unavailable
1076139 [main] sh 3448 fork: child -1 - forked process 12164 died unexpectedly, retry 0, exit code 0xC0000142, errno 11
D:\Program Files (x86)\Git\mingw32/libexec/git-core\git-submodule: fork: retry: Resource temporarily unavailable
3147113 [main] sh 3448 fork: child -1 - forked process 6652 died unexpectedly, retry 0, exit code 0xC0000142, errno 11
D:\Program Files (x86)\Git\mingw32/libexec/git-core\git-submodule: fork: retry: Resource temporarily unavailable
7205594 [main] sh 3448 fork: child -1 - forked process 16692 died unexpectedly, retry 0, exit code 0xC0000142, errno 11
D:\Program Files (x86)\Git\mingw32/libexec/git-core\git-submodule: fork: retry: Resource temporarily unavailable
15263777 [main] sh 3448 fork: child -1 - forked process 3892 died unexpectedly, retry 0, exit code 0xC0000142, errno 11
D:\Program Files (x86)\Git\mingw32/libexec/git-core\git-submodule: fork: Resource temporarily unavailable

Debug your Azure website remotely.

Site name: sharplab-edge (https://edge.sharplab.io)

[davidebbo]

Not sure what's going om here, but see http://blogs.suwatbodin.com/2013/07/deploy-to-windows-azure-using-git-with.html for some general guidance related to using submodules .

[davidebbo]

Also, I was not able to repro this. I tried this:

• Go to Kudu console on a test site
• In some test folder, run git clone https://github.com/ashmind/SharpLab.git
• Go in SharpLab folder and run git clone https://github.com/ashmind/SharpLab.git
• Now run git submodule sync. It just says Synchronizing submodule url for 'source/#external/ILSpy', with no errors.

[ashmind]

@davidebbo Thanks for the research! Yes, same code worked fine before, and still works fine on local. I googled the exit code from the error message (0xC0000142) and it seems to be "The application was unable to start correctly", which kind of makes sense -- git submodule sync tries to start something using fork, but it fails to start.

Not sure why it fails to start -- might be something with a specific system?
https://answers.microsoft.com/en-us/windows/forum/windows_10-performance-winpc/error-0xc0000142/3da27a75-178c-4de5-b43c-c538e6005540?auth=1

[davidebbo]

Can you try in other Web Apps to see if this seems specific to just this one app? That will help isolate. If it comes down to just one app, we can take a look at it and see what might be going on.

[ashmind]

@davidebbo Sure -- same happens to main sharplab (sharplab.io) deployments/SCM. Also tried sl-b-dotnet-features-dataflow through SCM (it doesn't have deployments) -- same issue when running git submodule sync in any folder.

[davidebbo]

So you mean that running git submodule sync fails this way in any folder that's not even a git repo? When I tried that in my app, I get:

fatal: Not a git repository (or any of the parent directories): .git

[ashmind]

Yep, surprising, but fail on 0xC0000142 in any directory -- probably because it fails before it can do any checks. Sorry, I have to leave for work, so further answers might be a bit delayed.

[davidebbo]

I moved you to a different VM, which solved it. But this is certainly puzzling. If you see this again, let me know and we'll dig deeper next time.

[ashmind]

Thanks! I'll definitely let you know if I see it again.

[davidebbo]

Ok, closing for now but we can reopen as needed.

[CanisLupus]

Sorry to comment in a closed issue, but we also had this problem in Microsoft Azure since last friday (for no apparent reason), which uses Kudu. We were pushing with git and the server was receiving the commits correctly, but then auto deployment via git never worked and didn't update our website. It's still failing, actually, but we uploaded manually via FTP.

We were getting errors when "git push"ing:

Counting objects: 25, done.
Delta compression using up to 16 threads.
Compressing objects: 100% (22/22), done.
Writing objects: 100% (25/25), 413.92 KiB | 8.12 MiB/s, done.
Total 25 (delta 12), reused 0 (delta 0)
remote:       2 [main] sh 7056 fork: child -1 - forked process 6500 died unexpectedly, retry 0, exit code 0xC0000142, errno 11
remote: hooks/post-receive: fork: retry: Resource temporarily unavailable
remote: 1094002 [main] sh 7056 fork: child -1 - forked process 6472 died unexpectedly, retry 0, exit code 0xC0000142, errno 11
remote: hooks/post-receive: fork: retry: Resource temporarily unavailable
remote: 3180523 [main] sh 7056 fork: child -1 - forked process 7564 died unexpectedly, retry 0, exit code 0xC0000142, errno 11
remote: hooks/post-receive: fork: retry: Resource temporarily unavailable
remote: 7246861 [main] sh 7056 fork: child -1 - forked process 4892 died unexpectedly, retry 0, exit code 0xC0000142, errno 11
remote: hooks/post-receive: fork: retry: Resource temporarily unavailable
remote: 15312972 [main] sh 7056 fork: child -1 - forked process 6948 died unexpectedly, retry 0, exit code 0xC0000142, errno 11
remote: hooks/post-receive: fork: Resource temporarily unavailable
To https://test-server-website.scm.azurewebsites.net:443/test-server-website.git
   9097984..498db46  master -> master

I don't know if this relates to Kudu, but I thought I would let you know. This link also has more details of the same issue, and someone seems to have narrowed it to sh.exe failing.

In our case, following these instructions yields the error, without even requiring git commands:

Go to Kudu Console. e.g. https://{yourapp}.scm.azurewebsites.net/DebugConsole
Run: echo ls | sh

[davidebbo]

@CanisLupus it could be that your VM got into that same unexplained state. If you share your site name, and can take a look.

Or if you need this fix asap, just scale up and back down to move VM (e.g. Small -> Medium -> Small, or vice versa).

[CanisLupus]

Hey, David, thanks for replying. I just created a new App with the name "kudu-test". The real site is the one that begins with "c" and ends in "www".

Unfortunately, we tried the option of scaling up and down before (waiting a few minutes between the two), and it didn't fix the problem, though we might have done something incorrectly. We managed to update our website manually by uploading everything via ftp.

Is it just a matter of going to "Scale up" and moving, for example, from S1 to S2 and then back again?

Thanks for looking into this. Daniel

[davidebbo]

@CanisLupus sorry, just saw this. Not sure why scale up / down didn't fix it. I initiated a reboot and now it seems fine.

[CanisLupus]

Hey @davidebbo, we're eternally grateful to you. ;) Git deployment is working fine now. Thank you so much!

[davidebbo]

@CanisLupus Glad that worked. It's a very strange condition with the git tools, and it is not well understood.