EACCES instead of ENOENT on EL6

rvagg commented 10 years ago

CentOS 6.5 to be specific (I can spin you up a DigitalOcean box if someone on core wants to tackle this). Kernel 2.6.32-431.1.2.0.1.el6.x86_64.

This comes from failures in the Node test suite and are presumably bubbling up from libuv but there are no equivalent tests here to demonstrate it. These tests are all failing because of unexpected EACCES error codes:

test/simple/test-child-process-exec-error
test/simple/test-child-process-spawn-error
test/simple/test-child-process-spawnsync
and even test/simple/test-cluster-eaccess

SELinux is "disabled" according to /etc/selinux/config

Fine on later Linuxes, haven't tested on EL7 yet though.

txdv commented 10 years ago

Who is on core? People who contribute a lot or have direct access to the repo?

What is EL6/EL7?

saghul commented 10 years ago

Who is on core? People who contribute a lot or have direct access to the repo?

https://github.com/joyent/libuv/blob/master/CONTRIBUTING.md

What is EL6/EL7?

RedHat's enterprise linux release.

rvagg commented 10 years ago

Sorry, not meaning to be too exclusive here, if somebody else is inclined to jump in and have a go but needs resources then I could probably spin up a dedicated DigitalOcean box just for you to mess around with this on. I'm mainly interested in getting basic CI targets happy and this is a hold-up for Node on EL6.

And as @saghul said, EL is "Enterprise Linux", although it's a bit broader than RHEL these days cause it includes CentOS, Oracle Linux and a few others, perhaps also encompassing Amazon Linux although that's diverged somewhat.

txdv commented 10 years ago

@saghul There is no definition of "core" in that readme.

saghul commented 10 years ago

core == project maintainers == people with commit acess. I'd also like my bike shed orange.

txdv commented 10 years ago

What does your bike shed have to do with this?

txdv commented 10 years ago

I have installed CentOS 7.0 and I get following test fail:

[bentkus@localhost libuv]$ test/run-tests
[%   0|+   0|-   0|T   0|S   0]: platform_output
Output from process `platform_output`:
uv_get_process_title: /home/bentkus/libuv/test/.libs/lt-run-tests
uv_resident_set_memory: 790528
uv_uptime: 860.000000
uv_getrusage:
  user: 0 sec 1169 microsec
  system: 0 sec 1169 microsec
uv_cpu_info:
  model: Intel(R) Core(TM) i3-3220T CPU @ 2.80GHz
  speed: 2787
  times.sys: 298400
  times.user: 456400
  times.idle: 7535400
  times.irq: 100
  times.nice: 0
uv_interface_addresses:
  name: lo
  internal: 1
  physical address: 00:00:00:00:00:00
  address: 127.0.0.1
  netmask: 255.0.0.0
  name: enp0s3
  internal: 0
  physical address: 08:00:27:d1:da:52
  address: 192.168.0.83
  netmask: 255.255.255.0
  name: lo
  internal: 1
  physical address: 00:00:00:00:00:00
  address: ::1
  netmask: ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff
  name: enp0s3
  internal: 0
  physical address: 08:00:27:d1:da:52
  address: fe80::a00:27ff:fed1:da52
  netmask: ffff:ffff:ffff:ffff::
=============================================================
[%  15|+  38|-   0|T   0|S   0]: tcp_writealot
`tcp_writealot` failed: exit code 6
Output from process `tcp_writealot`:
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
uv_write error: ECANCELED - operation canceled
Output from process `tcp_writealot`:
GOT EOF
Assertion failed in test/test-tcp-writealot.c on line 170: bytes_received_done == TOTAL_BYTES
=============================================================

@rvagg If you spin that machine up and give me the login, I will take a look at it.

rvagg commented 10 years ago

yeah, tcp_writealot seems to fail some of the time for me too. @txdv your public github keys are in the root account @ 104.131.148.215, just make yourself a user account to play.

If anybody else would like access then let me know. I don't want to leave this running for too long obviously but you can thank DigitalOcean for their sponsorship and support of Node & libuv here.

txdv commented 10 years ago

[bentkus@libuv-issue-1527-centos65 libuv]$ uname -a
Linux libuv-issue-1527-centos65 2.6.32-431.1.2.0.1.el6.x86_64 #1 SMP Fri Dec 13 13:06:13 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

[bentkus@libuv-issue-1527-centos65 libuv]$ test/run-tests
[%   0|+   0|-   0|T   0|S   0]: platform_output
Output from process `platform_output`:
uv_get_process_title: /home/bentkus/libuv/test/.libs/lt-run-tests
uv_resident_set_memory: 790528
uv_uptime: 35919.000000
uv_getrusage:
  user: 0 sec 999 microsec
  system: 0 sec 0 microsec
uv_cpu_info:
  model: Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz
  speed: 2399
  times.sys: 125700
  times.user: 462400
  times.idle: 358429500
  times.irq: 0
  times.nice: 0
  model: Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz
  speed: 2399
  times.sys: 75900
  times.user: 215400
  times.idle: 358790800
  times.irq: 0
  times.nice: 0
uv_interface_addresses:
  name: lo
  internal: 1
  physical address: 00:00:00:00:00:00
  address: 127.0.0.1
  netmask: 255.0.0.0
  name: eth0
  internal: 0
  physical address: 04:01:2c:37:4c:01
  address: 104.131.148.215
  netmask: 255.255.240.0
  name: lo
  internal: 1
  physical address: 00:00:00:00:00:00
  address: ::1
  netmask: ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff
  name: eth0
  internal: 0
  physical address: 04:01:2c:37:4c:01
  address: fe80::601:2cff:fe37:4c01
  netmask: ffff:ffff:ffff:ffff::
=============================================================
[% 100|+ 238|-   0|T   0|S   0]: Done.

No errors on that machine...

rvagg commented 10 years ago

these are fails in node's test suite that (very likely) bubble up from libuv and there are no equivalent libuv tests that cover this; see original issue description

txdv commented 10 years ago

A, ok, i'll test node then. I overlooked that part when you specified node.

txdv commented 10 years ago

I created the user bentkus, added the same ssh authorized_keys file that root has, so you can directly log the user.

I pulled joyent/node/master and ran the tests and I got only one failure:

[bentkus@libuv-issue-1527-centos65 node]$ make test
make -C out BUILDTYPE=Release V=1
make[1]: Entering directory `/home/bentkus/node/out'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/home/bentkus/node/out'
ln -fs out/Release/node node
/usr/bin/python tools/test.py --mode=release simple message
=== release test-v8-gc ===                                                     
Path: simple/test-v8-gc
Command: out/Release/node --expose_gc /home/bentkus/node/test/simple/test-v8-gc.js
--- CRASHED ---
[02:39|% 100|+ 776|-   1]: Done                                                
make: *** [test] Error 1

Seems unrelated to your errors. I ran the tests that you listed specifically and they just passed.

I don't know, maybe you are doing something different... Maybe setting it up differently?

saghul commented 10 years ago

Have you run it as root?

rvagg commented 10 years ago

try the v0.12 branch

txdv commented 10 years ago

[bentkus@libuv-issue-1527-centos65 node]$ make test
make -C out BUILDTYPE=Release V=1
make[1]: Entering directory `/home/bentkus/node/out'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/home/bentkus/node/out'
ln -fs out/Release/node node
/usr/bin/python tools/test.py --mode=release simple message
=== release test-crypto-domains ===                                            
Path: simple/test-crypto-domains
/home/bentkus/node/test/simple/test-crypto-domains.js:42
    throw new Error('pseudoRandomBytes');
          ^
Error: pseudoRandomBytes
    at Object.ondone (/home/bentkus/node/test/simple/test-crypto-domains.js:42:11)
Command: out/Release/node /home/bentkus/node/test/simple/test-crypto-domains.js
=== release test-v8-gc ===                                                     
Path: simple/test-v8-gc
Command: out/Release/node --expose_gc /home/bentkus/node/test/simple/test-v8-gc.js
--- CRASHED ---
[02:44|% 100|+ 785|-   2]: Done                                                
make: *** [test] Error 1
[bentkus@libuv-issue-1527-centos65 node]$ git branch
  master
* v0.12

Some crypto failure.

Can you maybe connect directly with the user bentkus and checkout the right commit you are having trouble with? Compile and run and check out the failures for maximum insurance?

rvagg commented 10 years ago

[bentkus@libuv-issue-1527-centos65 node]$ git branch
  master
* v0.12
[bentkus@libuv-issue-1527-centos65 node]$ make test-simple
make -C out BUILDTYPE=Release V=1
make[1]: Entering directory `/home/bentkus/node/out'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/home/bentkus/node/out'
ln -fs out/Release/node node
/usr/bin/python tools/test.py simple
=== release test-child-process-exec-error ===                                  
Path: simple/test-child-process-exec-error
child_process.js:1143
    throw errnoException(err, 'spawn');
          ^
Error: spawn EACCES
    at exports._errnoException (util.js:746:11)
    at ChildProcess.spawn (child_process.js:1143:11)
    at exports.spawn (child_process.js:976:9)
    at exports.execFile (child_process.js:670:15)
    at test (/home/bentkus/node/test/simple/test-child-process-exec-error.js:29:3)
    at Object.<anonymous> (/home/bentkus/node/test/simple/test-child-process-exec-error.js:46:1)
    at Module._compile (module.js:460:26)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
Command: out/Release/node /home/bentkus/node/test/simple/test-child-process-exec-error.js
=== release test-child-process-spawn-error ===                            
Path: simple/test-child-process-spawn-error
child_process.js:1143
    throw errnoException(err, 'spawn');
          ^
Error: spawn EACCES
    at exports._errnoException (util.js:746:11)
    at ChildProcess.spawn (child_process.js:1143:11)
    at exports.spawn (child_process.js:976:9)
    at Object.<anonymous> (/home/bentkus/node/test/simple/test-child-process-spawn-error.js:31:19)
    at Module._compile (module.js:460:26)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:501:10)
    at startup (node.js:132:16)
Command: out/Release/node /home/bentkus/node/test/simple/test-child-process-spawn-error.js
=== release test-child-process-spawnsync ===                          
Path: simple/test-child-process-spawnsync
sleep started
sleep exited [ 1, 4339366 ]
assert.js:98
  throw new assert.AssertionError({
        ^
AssertionError: "EACCES" === "ENOENT"
    at Object.<anonymous> (/home/bentkus/node/test/simple/test-child-process-spawnsync.js:48:8)
    at Module._compile (module.js:460:26)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:501:10)
    at startup (node.js:132:16)
    at node.js:850:3
Command: out/Release/node /home/bentkus/node/test/simple/test-child-process-spawnsync.js
=== release test-v8-gc ===                                                     
Path: simple/test-v8-gc
Command: out/Release/node --expose_gc /home/bentkus/node/test/simple/test-v8-gc.js
--- CRASHED ---
[02:43|% 100|+ 764|-   4]: Done                                                
make: *** [test-simple] Error 1
[bentkus@libuv-issue-1527-centos65 node]$

Ignore the crypto-domains bug, it's been dealt with elsewhere and is only an intermittent problem with the tests due to worker threads finishing at different times.

rvagg commented 10 years ago

oh! you know what this is? it only happens when you're root and su to a user account to run the tests. It just occured to me to ssh in as the user account and run the tests and now they are running without showing EACCES errors.

Mind you, I don't understand why EL5 would be doing su differently that would cause this but at least I can work around the problem.

@saghul & others: any thoughts on whether the difference is worth being concerned about? It means that behaviour will be modified when coming from root to a user account on these older linuxes, it'd likely even happen when a process is launched from an init script and perhaps even upstart (EL6 uses upstart, EL5 uses init). Is the lack of consistency a problem for libuv or are you happy to hand off blame to the OS and be done with it?

txdv commented 10 years ago

Looks like an issue with su or the os to me. We are basically calling just systems calls, I don't think we can do much about this in libuv.

bnoordhuis commented 10 years ago

It's unrelated to libuv but as a FYI, the simple/test-crypto-domains failure is a bug in the test that's been fixed in node-forward/node@1f11983 (currently private; if you want access, ping me.) Not sure why simple/test-v8-gc crashes.

I bet that the EACCESS error is because part of $PATH is inaccessible to the non-root user, i.e. that it's a bad assumption in the test.

saghul commented 10 years ago

Ok, closing then!

joyent / libuv

EACCES instead of ENOENT on EL6 #1527