Fail to boot when system is based on musl

[piranna]

I'm trying to use NodeOS on vagga. NodeOS is a musl-based operating system so it uses it's own ld-linux instead the usual one from glibc, like the one used on Ubuntu. For that reason, when executing vagga run it exits with error 127. Seems it's needed to exec the commands not only defining the LD_LIBRARY_PATH environment variable to use the vagrant lib folder (that seems it's being done correctly) but also using the correct ld-loader:

[piranna@Mabuk:~/Proyectos/NodeOS/node_modules/nodeos-barebones/.vagga/barebones]
 (vagga) > LD_LIBRARY_PATH=lib lib/libc.so bin/node
> 

[tailhook]

As far as I know it's handled by linux not by vagga. In particular the "interpreter" path has usually value hard-coded into binary itself.

Here it is on NixOS (the last line is intepreter):

$ ldd $(which node)
        linux-vdso.so.1 (0x00007ffc2cf94000)
        libz.so.1 => /nix/store/31w31mc8immhpnmxvcl4l0fvc3i5iwh0-zlib-1.2.8/lib/libz.so.1 (0x00007f59cad93000)
        libhttp_parser.so => /nix/store/k40zmqkmc8v61a4gj938z6xapnc3ijn2-http-parser-2.5.0/lib/libhttp_parser.so (0x00007f59cab8d000)
        libuv.so.1 => /nix/store/npa67ryxavxs1rjid62wk81sbq10d527-libuv-1.7.5/lib/libuv.so.1 (0x00007f59ca96b000)
        libcrypto.so.1.0.0 => /nix/store/arbd09p2cvj876rnymn9v7ziks4r2cb1-openssl-1.0.2d/lib/libcrypto.so.1.0.0 (0x00007f59ca517000)
        libssl.so.1.0.0 => /nix/store/arbd09p2cvj876rnymn9v7ziks4r2cb1-openssl-1.0.2d/lib/libssl.so.1.0.0 (0x00007f59ca29e000)
        libdl.so.2 => /nix/store/hd6km3hscbgl2yw8nx7lr5z9s8h89p04-glibc-2.21/lib/libdl.so.2 (0x00007f59ca09a000)
        librt.so.1 => /nix/store/hd6km3hscbgl2yw8nx7lr5z9s8h89p04-glibc-2.21/lib/librt.so.1 (0x00007f59c9e92000)
        libstdc++.so.6 => /nix/store/i9nn1fkcy95dzf0hb9wi8gbkid3iw1sa-gcc-4.9.3/lib/libstdc++.so.6 (0x00007f59c9b87000)
        libm.so.6 => /nix/store/hd6km3hscbgl2yw8nx7lr5z9s8h89p04-glibc-2.21/lib/libm.so.6 (0x00007f59c9884000)
        libgcc_s.so.1 => /nix/store/hd6km3hscbgl2yw8nx7lr5z9s8h89p04-glibc-2.21/lib/libgcc_s.so.1 (0x00007f59c966e000)
        libpthread.so.0 => /nix/store/hd6km3hscbgl2yw8nx7lr5z9s8h89p04-glibc-2.21/lib/libpthread.so.0 (0x00007f59c9451000)
        libc.so.6 => /nix/store/hd6km3hscbgl2yw8nx7lr5z9s8h89p04-glibc-2.21/lib/libc.so.6 (0x00007f59c90b1000)
        libnsl.so.1 => /nix/store/hd6km3hscbgl2yw8nx7lr5z9s8h89p04-glibc-2.21/lib/libnsl.so.1 (0x00007f59c8e99000)
        /nix/store/hd6km3hscbgl2yw8nx7lr5z9s8h89p04-glibc-2.21/lib/ld-linux-x86-64.so.2 (0x00007f59cafab000)

Here it is on Alpine, which is also musl-libc based (here it is first line for some reason):

# ldd $(which node)
        /lib/ld-musl-x86_64.so.1 (0x56204e71a000)
        libz.so.1 => /lib/libz.so.1 (0x7f2f423fb000)
        libcares.so.2 => /usr/lib/libcares.so.2 (0x7f2f421eb000)
        libuv.so.0.10 => /usr/lib/libuv.so.0.10 (0x7f2f41fca000)
        libssl.so.1.0.0 => /lib/libssl.so.1.0.0 (0x7f2f41d65000)
        libcrypto.so.1.0.0 => /lib/libcrypto.so.1.0.0 (0x7f2f41982000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x7f2f41679000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7f2f41467000)
        libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x56204e71a000)

Here it is on Ubuntu:

# ldd $(which node)
        linux-vdso.so.1 =>  (0x00007ffd21b5d000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd3216d6000)
        libcares.so.2 => /usr/lib/x86_64-linux-gnu/libcares.so.2 (0x00007fd3214c5000)
        libv8.so.3.14.5 => /usr/lib/libv8.so.3.14.5 (0x00007fd320eba000)
        libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007fd320c5b000)
        libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007fd320880000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd32067b000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fd320473000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fd32016f000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd31ff50000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd31fb8b000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd31f885000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fd31f66e000)
        /lib64/ld-linux-x86-64.so.2 (0x0000562681820000)

To run nixos binary on ubuntu you may use patchelf utility:

patchelf --set-interpreter /lib64/ld-linux-x86-64.so.2 node

Or vica versa. I've not tried to patch it to musl-based interpreter (I believe it may crash because of some binary incompatiblities).

Anyway I think you run the node binary from other system and compiled with different interpreter path.

[tailhook]

And well, about your example:

[piranna@Mabuk:~/Proyectos/NodeOS/node_modules/nodeos-barebones/.vagga/barebones]
 (vagga) > LD_LIBRARY_PATH=lib lib/libc.so bin/node
> 

This doesn't work, because interpreter paths are usually absolute. It should work if you run vagga:

vagga _run barebones /bin/node

Or if you at least chroot into the folder:

sudo chroot .vagga/barebones /bin/node

FYI, vagga runs Alpine linux (which is also musl-based) distribution just as well as ubuntu.

[piranna]

Anyway I think you run the node binary from other system and compiled with different interpreter path.

Yes, that was the problem, I had on Ubuntu installed globally musl 1.1.5 and was using on NodeOS 1.1.11, so seems get_nprocs (that was the missing symbol, by the way) was added between them :-P. I've desinstalled the globally one and it directly tells me that can't find the dynamic loader, so this was the problem.

(I've found this just while writting this answer... :-P)

This doesn't work, because interpreter paths are usually absolute

This was just for testing, to set the location NodeOS libraries and the dynamic loader were available. It works because instead of using the loader defined in the Node.js binary, you are using directly the loader lib/libc.so. It's somewhat similar to how shebangs works.

It should work if you run vagga:

vagga _run barebones /bin/node

Or if you at least chroot into the folder:

sudo chroot .vagga/barebones /bin/node

Amazing, it does! :-D In fact, they works when using /init, too :-) So now the point is, why vagga run doesn't work instead? :-/ My vagga.yaml file is

containers:
  barebones:
    setup:
    - !Tar
      url: http://localhost:8080/out/nocona/initramfs.tar

commands:
  run: !Command
    description: Start NodeOS barebones layer
    container: barebones
    pid1mode: exec
    run: /init

[tailhook]

Amazing, it does! :-D In fact, they works when using /init, too :-) So now the point is, why vagga run doesn't work instead? :-/

It should work. Can you run RUST_LOG=debug vagga run?

BTW, pid1mode: exec doesn't work at the moment (i.e. pid1mode is ignored). It used to work in vagga 0.1, but I never had a use case for that, so it was not implemented since vagga 0.2.

[tailhook]

Ah, I got it. When you supply a string in run you get it run by shell:

run: /init

Is translated to /bin/sh -c "/init"

To avoid using shell you need to set a list:

run: ["/init"]

[piranna]

Ah, I got it. When you supply a string in run you get it run by shell:

run: /init

Is translated to /bin/sh -c "/init"

To avoid using shell you need to set a list:

run: ["/init"]

[piranna@Mabuk:~/Proyectos/NodeOS/node_modules/nodeos-barebones]
 (vagga) > vagga run
Connecting to localhost:8080 (127.0.0.1:8080)
68d8e510-barebones.t 100% |*****************************************************| 12912k  0:00:00 ETA
> var os = require('os')
undefined
> os.nodeos
'0.0.0'

Profit! :-D

BTW, pid1mode: exec doesn't work at the moment (i.e. pid1mode is ignored). It used to work in vagga 0.1, but I never had a use case for that, so it was not implemented since vagga 0.2.

Well, I know in normal conditions is not necesary, but as the documentation says, for situations when it's being launched an init-like process it would make sense... It doesn't need to be too complicated, NodeOS use century, that's just a safety-net for zombie process and delegate all it's functionality to another binary (maybe vagga PID1 is working this way?). I don't think it's something urgent or needed at all, but maybe would be a useful option to have...

[tailhook]

maybe vagga PID1 is working this way?

Yes exactly.

I don't think it's something urgent or needed at all, but maybe would be a useful option to have...

Sure. Feel free to open issue for that.

[piranna]

maybe vagga PID1 is working this way?

Yes exactly.

Oh cool, good to know, this way on vagga century would not be needed :-)

I don't think it's something urgent or needed at all, but maybe would be a useful option to have...

Sure. Feel free to open issue for that.

Ok, I've just done it