Musl dynamic linked binary use glibc dynamic linker (not musl)

[bjornfor]

Issue description

musl has it's own dynamic linker, but it seems that builds with Nix/nixpkgs ends up using the glibc linker. Building with musl-gcc on Ubuntu (no Nix) uses the correct (musl) linker.

Steps to reproduce

{ pkgs ? import <nixpkgs> {} }:

with pkgs;

stdenv.mkDerivation rec {
  name = "musl-test";
  buildInputs = [ musl ];
  buildCommand = ''
    mkdir -p "$out/bin"

    cat >> main.c << EOF
    #include <stdio.h>
    int main(int argc, char *argv[])
    {
        printf("Hello Musl world\n");
    }
    EOF

    musl-gcc main.c -o "$out/bin/musl-dynamic-test"
    musl-gcc -static main.c -o "$out/bin/musl-static-test"
  '';
}

$ file $(nix-build test.nix)/bin/musl-dynamic-test
/nix/store/7kvcmxzv8vcwdjpba7cs9hwncdjdgly2-musl-test/bin/musl-dynamic-test: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/7crrmih8c52r8fbnqb933dxrsp44md93-glibc-2.25/lib/ld-linux-x86-64.so.2, not stripped

In the above output, I expect to see "musl", not "glibc".

Technical details

• System: NixOS 17.03

[bjornfor]

musl has a .spec file that lists the correct dynamic linker ("/nix/store/...-musl-1.1.15/lib/ld-musl-x86_64.so.1"). Adding NIX_DEBUG=1 to the build shows that someone passes / overrides the -dynamic-linker option to use glibc.

I guess a whole separate stdenv for musl needs to be created to fix this?

[copumpkin]

I guess a whole separate stdenv for musl needs to be created to fix this?

Yes, I'd expect so. I'd love to see that happen, though. I think someone was talking about that recently on IRC. Was it @cleverca22?

[copumpkin]

Also, normally I'd be a bit concerned about "forking" behavior for such a musl stdenv, but we already support two "libc"s right now: glibc and Darwin's libSystem/libc aggregate, so adding a third doesn't seem ridiculous to me, as long as we can get some hydra jobs in and people commit to keeping it healthy.

[ghost]

I also have this issue in a much more hacky environment I'm working in.

( I wanted to compile something with musl so that I could get the size into the couple tens of kilobytes range, but it would bloat up to glibc sizes and I managed to confirm that nix was passing a lot of gcc stuff in. Good to know about NIX_DEBUG=1 even if later. )

[ghost]

A temporary hack that seems to work is to pass the path to your GCC via the REALGCC env var, this is used by musl's musl-gcc wrapper. So you can do something like REALGCC=$(grep -Eo '".+/gcc"' $(which gcc) | tr -d '"') musl-gcc something.c

I don't know what this will break.

EDIT: (In case it wasn't clear, I meant pass the path to the raw GCC executable and not the wrapper)

[vcunat]

First, stdenv (in particular the wrapped compiler inside) has knowledge of the loader it wants to use, so you can't just use our stock glibc stdenv. Second, our wrapper would also need fixes.

With https://github.com/vcunat/nixpkgs/commit/bcc0f8045 I can do this

{ pkgs ? import ./. {} }:

with pkgs;
let
  stdenv-musl = overrideCC stdenv gcc-musl;
  gcc-musl = wrapCCWith {
    name = "gcc+musl";
    cc = stdenv.cc.cc;
    libc = musl;
    detectLD = true;
  };
in

stdenv-musl.mkDerivation rec {
  name = "musl-test";
  buildCommand = ''
    mkdir -p "$out/bin"

    cat >> main.c << EOF
    #include <stdio.h>
    int main(int argc, char *argv[])
    {
        printf("Hello Musl world\n");
    }
    EOF

    cc main.c -o "$out/bin/musl-dynamic-test"
    cc -static main.c -o "$out/bin/musl-static-test"
  '';
}

(Beware that wrapCCWith has changed significantly not too long ago.)

/cc @Ericson2314 about ideas on the best way to replace my hack-commit.

[Ericson2314]

Err aren't we always detecting the loader today, and in your example? I don't see why libc = musl with wrapCCWith already doesn't work.

[vcunat]

@Ericson2314: it doesn't work because the wrapper is hardcoded to ld-linux-x86-64.so.2 on x86_64-linux whereas musl uses a different name.

[Ericson2314]

Oh right. I misread that. I'd just add a targetPlatform.libc == "musl" to the chain.

[vcunat]

Ah, so you propose to handle libc change like cross-compilation. I expect that will be more practical when having more dependencies...

[dtzWill]

There's support for musl cross and native now :D, so I think this can be closed?

Also: https://github.com/NixOS/rfcs/pull/23

[Ericson2314]

Agreed. If the RFC passes, this is fixed for good, if the RFC fails (heaven forbid) this is won't-fix, so either way this is done.

[nh2]

For readers passing by: You can now easily try musl-linked C programs with nix by using the musl-gcc wrapper that's available in nixpkgs master > 18.03 (e.g. using the commit pinned below):

$ NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-shell -p musl
[nix-shell:]$ musl-gcc myprogram.c -o myprogram

EDIT: This is wrong, see https://github.com/NixOS/nixpkgs/issues/25178#issuecomment-423003570 below for how to do it right.

[bjornfor]

I'm on NixOS 18.03, but using your pinned nixpkgs example (master > 18.03) should still work, right? However, it does not work for me:

$ NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-shell -p musl
$ musl-gcc /tmp/hello.c
$ file a.out
a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/ld-linux-x86-64.so.2, with debug_info, not stripped

IOW, I'm still getting glibc into my musl. :-)

[nh2]

@bjornfor You are right, a colleague pointed out today too that indeed glibc sneaks in. I now know why:

Looks like even musl-gcc calls plain gcc which includes the nix's stdenv flags which target glibc stuff. Found with strace -fe execve musl-gcc ....

(-static makes those flags to be ignored, so building statically is one workaround.)

The following works, also without -static:

/nix/store/vv4r320p5yd1k01kld62q1lppjxcswhb-gcc-7.3.0/bin/gcc -specs /nix/store/j88hc0jy0h1g350a7n0g2m29yfvvsrrn-musl-1.1.20-dev/lib/musl-gcc.specs -g myprogram.c -lpthread -o myprogram

@bjornfor, try that!

This what musl-gcc eventually calls but with some flags removed that pass libc stuff. The full flags were:

/nix/store/vv4r320p5yd1k01kld62q1lppjxcswhb-gcc-7.3.0/bin/gcc -O2 -D_FORTIFY_SOURCE=2 -fstack-protector-strong --param ssp-buffer-size=4 -fno-strict-overflow -Wformat -Wformat-security -Werror=format-security -fPIC -Wl,-dynamic-linker -Wl,/nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/ld-linux-x86-64.so.2 -g myprogram.c -lpthread -o myprogram -specs /nix/store/j88hc0jy0h1g350a7n0g2m29yfvvsrrn-musl-1.1.20-dev/lib/musl-gcc.specs -B/nix/store/zk5zj2307zxaq7dx585yia3dn5k4qlsl-gcc-7.3.0-lib/lib -B/nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/ -idirafter /nix/store/akak0rxhbi4n87z3nx78ipv76frvj841-glibc-2.27-dev/include -idirafter /nix/store/vv4r320p5yd1k01kld62q1lppjxcswhb-gcc-7.3.0/lib/gcc/x86_64-unknown-linux-gnu/7.3.0/include-fixed -B/nix/store/iw94llkj05wgaz268mlzvgx8jkbi1ss0-gcc-wrapper-7.3.0/bin/ -isystem /nix/store/j88hc0jy0h1g350a7n0g2m29yfvvsrrn-musl-1.1.20-dev/include -isystem /nix/store/j88hc0jy0h1g350a7n0g2m29yfvvsrrn-musl-1.1.20-dev/include -Wl,-rpath -Wl,/nix/store/1kf4h2shw37n8n0k0zlmypfv3zmrbd03-shell/lib64 -Wl,-rpath -Wl,/nix/store/1kf4h2shw37n8n0k0zlmypfv3zmrbd03-shell/lib -L/nix/store/mbi77di3ndy5kq3j1a0420pwz5h11ywh-musl-1.1.20/lib -L/nix/store/mbi77di3ndy5kq3j1a0420pwz5h11ywh-musl-1.1.20/lib -L/nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib -L/nix/store/zk5zj2307zxaq7dx585yia3dn5k4qlsl-gcc-7.3.0-lib/lib

@dtzWill do you know what we can do so that these glibc related flags don't leak in via the gcc-wrapper?

[dtzWill]

Honestly I've never used the musl-gcc wrapper much-- it's really a quickfix at best and should never be used in the place of a proper musl-based toolchain. I actually disabled it originally, partially to avoid problems when people used it expecting it to act like what you get with setting crossSystem or localSystem appropriately. I've heard folks say it's useful in a pinch but I'm not sure it's worth debugging if you're running into problems.

The interface for generating a shell from a given 'stdenv' is not as simple as it should be, but something like the following should get you a 'cc' that builds against musl:

$ nix run nixpkgs.pkgsMusl.stdenv.cc
$ cc test.c -o test-c
$ file /test-c
./test-c: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /nix/store/35mb6imifzx9316y7nknhh6apx1k7a24-musl-1.1.19/lib/ld-musl-x86_64.so.1, with debug_info, not stripped

(-static should work at least for C programs, forget if things provide static versions of the c++ libraries or not)

Same caveats apply as normal re:using Nix-generated compiler wrappers and library dependencies, but this should get you started! Hope this helps!

[nh2]

I actually disabled it originally

Ah, that's why I couldn't find it in older commits. I thought its presence in newer commits meant that it's a new addition included for that purpose and that I should use it consequently.

[dtzWill]

Oh also keep in mind that for reasons that make no sense to me, 'nix-shell -p' actually unconditionally pulls in the default stdenv!

So, basically, I strongly recommend avoiding nix-shell -p when doing anything related to compilers or you'll run into all kinds of badness. This can be seen with something as simple as nix-shell -p gcc8 and checking cc --version.

(this is because it uses runCommand and "for legacy reasons" but ...well anyway, if you didn't know about this "fun fact" that might be what you're encountering as well)

[domenkozar]

That's because -p {pkgs} is just templating stdenv.mkDerivation { buildInputs = [ pkgs ... ]; }