Cannot spawn child process on azure emulator or real azure vms since node 0.8.0

[kessler]

server.js:

fork = require('child_process').fork;

fork(__dirname + "/test.js");
fork(__dirname + "/test.js");
fork(__dirname + "/test.js");

test.js

console.log('forked!');

When run on the same machine with normal node.exe process, this code executes fine, however when deployed to an azure emulator or a real azure vm with the same node.exe being used, code fails with the following: (node v0.8.0)

child_process.js:789
     throw errnoException(errno, 'spawn');
           ^
 Error: spawn Unknown system errno 6
     at errnoException (child_process.js:837:11)
     at ChildProcess.spawn (child_process.js:789:11)
     at exports.spawn (child_process.js:614:9)
     at exports.fork (child_process.js:443:10)
     at Object.<anonymous> (X:\deploy\local_package.csx\roles\z\approot\server.js:3:1)
     at Module._compile (module.js:449:26)
     at Object.Module._extensions..js (module.js:467:10)
     at Module.load (module.js:356:32)
     at Function.Module._load (module.js:312:12)
     at Module.runMain (module.js:487:10)

or on latest node (v0.8.4):

 child_process.js:789
     throw errnoException(errno, 'spawn');
           ^
 Error: spawn EBADF
     at errnoException (child_process.js:837:11)
     at ChildProcess.spawn (child_process.js:789:11)
     at exports.spawn (child_process.js:614:9)
     at exports.fork (child_process.js:443:10)
     at Object.<anonymous> (X:\spawn-child-process-bug\local_package.csx\roles\HelloWorker\approot\server.js:3:1)
     at Module._compile (module.js:449:26)
     at Object.Module._extensions..js (module.js:467:10)
     at Module.load (module.js:356:32)
     at Function.Module._load (module.js:312:12)
     at Module.runMain (module.js:492:10)

This occurs on both 32bit and 64bit flavors of node.

Here is a link to a git repo:

https://github.com/yanush/spawn-child-process-problem

containing a ready to deploy azure package with the code above, reproducing the problem.

run in your emulator or deploy to a real vm.

[bnoordhuis]

/cc @piscisaureus

[martinthomson]

I've encountered the same issue. The problem is not with the machine, since I can log in to the instance using remote desktop and run the process successful in a command line window.

I have a workaround, though it wont help with this specific test case. Since the error is with file descriptors, disabling stdin/stdout/stderr actually helps. I managed to work around this issue by passing the silent option to the fork:

fork(__dirname + "/test.js", [], { silent: true });

I have also captured a diff of process.env with and without the problem that can be provided on request.

[kessler]

Awesome, this is a good workaround for now...

[piscisaureus]

Node fails because it tries to make the forked worker inherit it's stdio handles, but node's stdio handles are not valid. Apparently node is being started by some process that doesn't set it up properly.

@tjanczuk @glennblock Any ideas why this is the case? Any chance you could fix azure :-)

[glennblock]

No idea, but there is a worker process which starts the entry point process in worker role which is where this is happening. We'll have to investigate.

On Mon, Jul 30, 2012 at 10:55 PM, Bert Belder < reply@reply.github.com

wrote:

Node fails because it tries to make the forked worker inherit it's stdio handles, but node's stdio handles are not valid. Apparently node is being started by some process that doesn't set it up properly.

@tjanczuk @glennblock Any ideas why this is the case? Any chance you could fix azure :-)

---

Reply to this email directly or view it on GitHub: https://github.com/joyent/node/issues/3779#issuecomment-7370643

[tjanczuk]

@martinthomson @gblock can you run an experiment and invoke node.exe in Azure by explicitly redirecting inputs and outputs, e.g.

node.exe server.exe 2>&1 > nul < nul

also try that with some file?

[piscisaureus]

@tjanczuk ^-- possible workaround

[kessler]

@tjanczuk is "server.exe" there a typo? because if it does, then I tried with "node.exe server.js 2>&1 > nul < nul" on the emulator and it didn't work (embedded it in the <ProgramEntryPoint commandLine="")

[tjanczuk]

Yes it was a typo, I meant server.js. Did you properly escape the command line (&amp; etc)?

[kessler]

Here is a direct copy paste from my ServiceDefinition.csdef:

<ProgramEntryPoint commandLine="node.exe .\server.js 2&gt;&amp;1 &gt; 1.txt &lt; 2.txt" setReadyOnProcessStart="true" />

p.s if you want, you can try it yourself on an emulator with almost zero work, just clone the git repo mentioned above, it has the barebone code needed to reproduce the problem and is "ready to deploy".

[glennblock]

Did it work when you tested @tjanczuk's reco?

On Wed, Aug 1, 2012 at 5:47 AM, yanush < reply@reply.github.com

wrote:

Here is a direct copy paste from my ServiceDefinition.csdef:

<ProgramEntryPoint commandLine="node.exe .\server.js 2&gt;&amp;1 &gt; 1.txt &lt; 2.txt" setReadyOnProcessStart="true" />

p.s if you want, you can try it yourself on an emulator with almost zero work, just clone the git repo mentioned above, it has the barebone code needed to reproduce the problem and is "ready to deploy".

---

Reply to this email directly or view it on GitHub: https://github.com/joyent/node/issues/3779#issuecomment-7412022

[kessler]

It did not... as was mentioned earlier :)

I wasn't sure if its the correct "syntax" though, since I speculated the cli might interpret it as node arguments rather than std redirects.

not sure how its suppose to discern node.exe bla.js a b c from node.exe bla.js 2>&1 > nul < nul

[piscisaureus]

@yanush Can you try if the node.exe from http://2bs.nl/node.zip works for you?

[glennblock]

There's a way to run elevated. I need to check what it is.

cab your try this

cmd.exe /k node.exe server.js

On 8/1/12, yanush reply@reply.github.com wrote:

It did not... was mentioned earlier :)

---

Reply to this email directly or view it on GitHub: https://github.com/joyent/node/issues/3779#issuecomment-7415185

[kessler]

just an FYI:

I tried <Runtime executionContext="elevated" ...> (its already included in the problem reproducing code) in ServiceDefinition.csdef, didn't help.

Will try that node.exe now.

[kessler]

@piscisaureus tried that exe now, got the same error (with and without the redirection suggested by @tjanczuk)

[glennblock]

Did you try running "cmd.exe /k node.exe server.js" that will spawn a new shell.

On Wed, Aug 1, 2012 at 9:41 AM, yanush < reply@reply.github.com

wrote:

just an FYI:

I tried &ltlRuntime executionContext="elevated" ...> (its already included in the problem reproducing code) in ServiceDefinition.csdef, didn't help.

Will try that node.exe now.

---

Reply to this email directly or view it on GitHub: https://github.com/joyent/node/issues/3779#issuecomment-7416099

[kessler]

Just tried that as well, it crashes the VM spawned by the emulator, I think because it doesn't have cmd.exe on path, i'm just checking that out now

[glennblock]

You probably have specify the fully qualified path or use environment variables to get to it.

On Wed, Aug 1, 2012 at 9:52 AM, yanush < reply@reply.github.com

wrote:

Just tried that as well, it crashes the VM the emulator spawns, I think because it doesn't have cmd.exe on path, i'm just checking that out now

---

Reply to this email directly or view it on GitHub: https://github.com/joyent/node/issues/3779#issuecomment-7416243

[glennblock]

You should be able to test in the emulator first.....

On Wed, Aug 1, 2012 at 9:55 AM, Glenn Block glenn.block@gmail.com wrote:

You probably have specify the fully qualified path or use environment variables to get to it.

On Wed, Aug 1, 2012 at 9:52 AM, yanush < reply@reply.github.com

wrote:

Just tried that as well, it crashes the VM the emulator spawns, I think because it doesn't have cmd.exe on path, i'm just checking that out now

---

Reply to this email directly or view it on GitHub: https://github.com/joyent/node/issues/3779#issuecomment-7416243

[glennblock]

Does it repro in the emulator btw?

On Wed, Aug 1, 2012 at 9:52 AM, yanush < reply@reply.github.com

wrote:

Just tried that as well, it crashes the VM the emulator spawns, I think because it doesn't have cmd.exe on path, i'm just checking that out now

---

Reply to this email directly or view it on GitHub: https://github.com/joyent/node/issues/3779#issuecomment-7416243

[kessler]

This occurs on both 32bit and 64bit flavors of node.

Here is a link to a git repo:

https://github.com/yanush/spawn-child-process-problem

containing a ready to deploy azure package with the code above, reproducing the problem.

run in your emulator or deploy to a real vm.

Yep :)

and also, managed to get the cmd.exe to work after welding the hard paths of cmd.exe, node.exe and server.js into the ServiceDefinition.csdef. It didn't solve the problem; I'm still getting EBADF.

[glennblock]

OK, so

1. Using elevation does not work.
2. Spawning a new shell does not work
3. Redirecting the output streams does not work.

right?

On Wed, Aug 1, 2012 at 9:57 AM, yanush < reply@reply.github.com

wrote:

This occurs on both 32bit and 64bit flavors of node.

Here is a link to a git repo:

https://github.com/yanush/spawn-child-process-problem

containing a ready to deploy azure package with the code above, reproducing the problem.

run in your emulator or deploy to a real vm.

Yep :)

and also, managed to get the cmd.exe to work after welding the hard path into the ServiceDefinition.csdef. It didn't solve the problem; I'm still get EBADF.

---

Reply to this email directly or view it on GitHub: https://github.com/joyent/node/issues/3779#issuecomment-7416323

[kessler]

I'm on #node.js atm if you want to make this dialog more direct.

[glennblock]

BTW, thanks for trying this stuff out :-)

On Wed, Aug 1, 2012 at 9:57 AM, yanush < reply@reply.github.com

wrote:

This occurs on both 32bit and 64bit flavors of node.

Here is a link to a git repo:

https://github.com/yanush/spawn-child-process-problem

containing a ready to deploy azure package with the code above, reproducing the problem.

run in your emulator or deploy to a real vm.

Yep :)

and also, managed to get the cmd.exe to work after welding the hard path into the ServiceDefinition.csdef. It didn't solve the problem; I'm still get EBADF.

---

Reply to this email directly or view it on GitHub: https://github.com/joyent/node/issues/3779#issuecomment-7416323

[kessler]

So far the only thing that worked is the silent: true workaround. So unless I did something wrong with my other experiments (which is always possible) all the other workarounds in your list, don't work.

[kessler]

Ohh and cheers, no thanks are needed :)

[glennblock]

Which "silent: true" workaround was that? I must have missed it.....

On Wed, Aug 1, 2012 at 10:03 AM, yanush < reply@reply.github.com

wrote:

So far the only thing that worked is the silent: true workaround. So unless I did something wrong with my other experiments (which is always possible) all the other workarounds in your list, don't work.

---

Reply to this email directly or view it on GitHub: https://github.com/joyent/node/issues/3779#issuecomment-7416406

[kessler]

first comment by @martinthomson on this thread (3rd box from the top)

[kessler]

Another important FYI

I tested this problem on all versions of node between 0.7.5 - 0.8.4, 0.7.9 version is the last one that has forking work on azure, 0.8.0 forking stops working. The error on 0.8.0 is a little bit different; its a number - 6 instead of EBADF.

[glennblock]

This is really important info. Thanks for all the investigation so far!

On Wed, Aug 1, 2012 at 10:19 AM, yanush < reply@reply.github.com

wrote:

Another important FYI

I tested this problem on all versions of node between 0.7.5 - 0.8.4, 0.7.9 version is the last one that has forking work on azure, 0.8.0 forking stops working. The error on 0.8.0 is a little bit different; its a number - 6 instead of EBADF.

---

Reply to this email directly or view it on GitHub: https://github.com/joyent/node/issues/3779#issuecomment-7416621

[glennblock]

@piscisaureus is it possible this is a regression introduced after node 0.7.9? Apparently it worked before then.

[piscisaureus]

@yanush the difference between "UNKNOWN system errno 6" and EBADF is not really relevant - this is exactly the same error.

[kessler]

@piscisaureus yup I know, merely stated it as evidence to the "detailed" analysis I've made :)

[piscisaureus]

Unfortunately I can't really debug this, because the azure sdk won't install on my machine (the installer just hangs).

[kessler]

I can setup a remote VM for you to do as you like... how about that?

[glennblock]

Thanks Yanush

I had skyped with Bert and I am setting up Bert a box with the SDK that he can remote into. I had a VM already setup with VS (he needs the debugger) so it's quick for me.

Thanks Glenn

[kessler]

Good news, I would greatly appreciate it if you keep me in the loop :)

[glennblock]

Yanush, you said the error occurs in the emulator as well? Bert tried and doesn't see that....

On Thu, Aug 2, 2012 at 12:07 AM, yanush < reply@reply.github.com

wrote:

Good news, I would greatly appreciate it if you keep me in the loop :)

---

Reply to this email directly or view it on GitHub: https://github.com/joyent/node/issues/3779#issuecomment-7431598

[kessler]

https://github.com/yanush/spawn-child-process-problem/blob/master/emulator.png

Did u use the same deployment I used ? (taken from the git repo)

[kessler]

Also @martinthomson encountered the same issue, though not sure if he ran into it in the emulator as well

[glennblock]

@bertbelder did you try this code?

On Thu, Aug 2, 2012 at 5:43 AM, Yaniv Kessler < reply@reply.github.com

wrote:

Also @martinthomson encountered the same issue, though not sure if he ran into it in the emulator as well

---

Reply to this email directly or view it on GitHub: https://github.com/joyent/node/issues/3779#issuecomment-7440972

[kessler]

However it just occurred to me that if @piscisaureus didn't ran into this then maybe its a problem with the deployment setup and he did something different. Mine own is based on HelloWorker from azure sdk...

@piscisaureus can u share your deployment settings (ServiceDefinition.csdef and the others) ?

[piscisaureus]

@yanush Yes I checked out your repo and ran in in the azure emulator. No problem

[kessler]

@piscisaureus that is bad news (for me)... now what do I do?

[piscisaureus]

@yanush http://yfrog.com/oejqgcp

[kessler]

@piscisaureus What version of azure sdk u have there?

[piscisaureus]

@yanush I am not saying there is no issue - I just couldn't reproduce it, which makes it hard to fix.

I am going to try with a real azure box first, I don't have a plan B yet.

[piscisaureus]

@yanush How do I find out? It was installed by @glennblock

[kessler]

Are you on irc by any chance?