musl: Trivial binaries built with musl will fault on nixos

[nagisa]

Describe the bug

Running statically linked musl libraries on nixos will fault in early runtime initialization. There appear to be multiple issues, first is that linking on nixos when PIE or PIC is enabled will implicitly also add the glibc ld-linux.so, which causes issues in rust binaries: https://github.com/rust-lang/rust/issues/74757, but it looks like trivial C binaries don’t work either...

This is the backtrace:

(gdb) bt
#0  0x00007ffff7f7aba6 in do_init_fini () from /nix/store/d8d7k50ffblx4dkqrg4j4flj41dxbwcp-musl-1.1.24/lib/libc.so
#1  0x00007ffff7f7bf43 in __libc_start_init () from /nix/store/d8d7k50ffblx4dkqrg4j4flj41dxbwcp-musl-1.1.24/lib/libc.so
#2  0x00007ffff7f3fc33 in libc_start_main_stage2 () from /nix/store/d8d7k50ffblx4dkqrg4j4flj41dxbwcp-musl-1.1.24/lib/libc.so
#3  0x0000000000401059 in _start ()

To Reproduce

Observe that building static binary works:

$ nix-shell -p 'musl' --run 'echo "int main(void) { return 0; }" | musl-gcc -static -xc - && ./a.out'

but if run with a dynamic interpreter, the binary will fault:

$ /nix/store/jx19wa4xlh9n4324xdl9rjnykd19mmq3-glibc-2.30/lib/ld-linux-x86-64.so.2 ./a.out
fish: “/nix/store/jx19wa4xlh9n4324xdl9…” terminated by signal SIGSEGV (Address boundary error)

Running a dynamically linked binary doesn’t work either (it encodes the same interpreter in its DT_INTERP elf metadata):

$ nix-shell -p 'musl' --run 'echo "int main(void) { return 0; }" | musl-gcc -xc - && ./a.out'
/run/user/1000/nix-shell-8331-0/rc: line 1:  8374 Segmentation fault      (core dumped) ./a.out

Expected behavior

Binaries linked against musl should work regardless of whether they are linked statically, dynamically or are position independent.

Screenshots N/A

Additional context N/A

Notify maintainers

@thoughtpolice @dtzWill

Metadata

• system: "x86_64-linux"
• host os: Linux 5.6.16, NixOS, 20.09pre228599.7a07f2a5edd (Nightingale)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3.6
• channels(root): "nixos-20.09pre228599.7a07f2a5edd"
• channels(nagisa): ""
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[nagisa]

As per #musl channel on IRC running binaries using the musl libc with the glibc interpreter is not supported.

[matthewbauer]

I think the Glibc-based stdenv from nix-shell -p is interfering with musl here. For instance, this works:

nix-shell -p musl --run 'echo "int main(void) { return 0; }" | NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu= musl-gcc -xc - && ./a.out'

The new Nix shell command in unstable nix doesn't automatically pull in stdenv:

nix shell nixpkgs#gcc nixpkgs#musl.dev -c sh -c 'echo "int main(void) { return 0; }" | musl-gcc -static -xc - && ./a.out'

[sternenseemann]

This also affects non static builds. The most obvious way to compile something with musl, i. e. musl-gcc test.c will result in a binary which segfaults, because it has the wrong interpreter set.

[nagisa]

The most obvious way to compile something with musl, i. e. musl-gcc test.c will result in a binary which segfaults

This was mentioned in the original report as well.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[harrysarson]

Not stale as far as I can tell :)

[stale[bot]]

I marked this as stale due to inactivity. → More info

[tv42]

Still a bug:

$ nix-shell -p 'musl' --run 'echo "int main(void) { return 0; }" | musl-gcc -xc - && ./a.out'
these paths will be fetched (1.73 MiB download, 9.86 MiB unpacked):
  /nix/store/782cjhv3rbf36d3gbbr3lj3vsp4bx5ly-musl-1.2.2
  /nix/store/jx3h50p252aw62k4j6x5r15r52l3nkkb-linux-headers-5.14
  /nix/store/qsjy464h808jmcgmya2ya9f49cj7y2fm-musl-1.2.2-dev
copying path '/nix/store/jx3h50p252aw62k4j6x5r15r52l3nkkb-linux-headers-5.14' from 'https://cache.nixos.org'...
copying path '/nix/store/782cjhv3rbf36d3gbbr3lj3vsp4bx5ly-musl-1.2.2' from 'https://cache.nixos.org'...
copying path '/nix/store/qsjy464h808jmcgmya2ya9f49cj7y2fm-musl-1.2.2-dev' from 'https://cache.nixos.org'...
/tmp/nix-shell-1039494-0/rc: line 1: 1039563 Segmentation fault      (core dumped) ./a.out

[osresearch]

It also seems to cause the normal gcc to link against musl libc by appending -L/nix/store/...-musl-1.2.3/lib to the command line?

nix-shell -p musl  --run 'echo "int main(void) { return 0; }" | gcc -xc - && ./a.out'
/tmp/nix-shell-24816-0/rc: line 3: 25571 Segmentation fault      (core dumped) ./a.out

% ldd a.out
    linux-vdso.so.1 (0x00007ffd07dd6000)
    libc.so => /nix/store/wjbx7jlcd7wxhgpp1ck4l14h30ijc2h3-musl-1.2.3/lib/libc.so (0x00007fb8203c0000)

My expectation was that gcc would produce glibc binaries and musl-gcc would wrap it so that the musl libc would be linked.

[jhheider]

For the specific use case of producing static binaries (with musl-clang), this seems to work:

sed -i.bak -e 's/-dynamic-linker "\$ldso"//' ld.musl-clang
sed -i.bak -e 's/^sflags=$/sflags="-static"/' musl-clang