search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.41k stars 13.63k forks source link

stage-final-gcc: Cross-compiled libgcc_s.so.1 retains debug info #243642

Open jonathan-conder opened 1 year ago

jonathan-conder commented 1 year ago

Describe the bug

When cross compiling some packages I noticed they had some unexpected dependencies like linux kernel headers. Tracked this down to libgcc_s.so.1 which has references to the dev output of glibc. Looks like it needs to be stripped.

Steps To Reproduce

Steps to reproduce the behavior (on x86_64-linux):

$ nixpkgs=nixpkgs/45ae0efbbce2aada6d5e8de6ace0c803b08ac9c7
$ pkg=pkgsCross.armv7l-hf-multiplatform.stdenv.cc.cc

$ nix build "$nixpkgs#$pkg"
$ libgcc="$(nix eval --raw "$nixpkgs#$pkg.libgcc.outPath")"
$ libgcc_s="$libgcc/lib/libgcc_s.so.1"

$ file "$libgcc_s"
/nix/store/kgd4n1y4yf1xni5kfdfcp6qx4wc5n3f4-armv7l-unknown-linux-gnueabihf-stage-final-gcc-12.3.0-libgcc/lib/libgcc_s.so.1: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, with debug_info, not stripped

$ strings "$libgcc_s" | grep nix
/nix/store/qq4xas5iyvvcrvd6r9h3727hw9cfzk2b-glibc-armv7l-unknown-linux-gnueabihf-2.37-8-dev/include
...

$ objdump -sj .debug_line "$libgcc_s" | grep nix
...
 01a10 63002f6e 69782f73 746f7265 2f717134  c./nix/store/qq4
...

Expected behavior

The library should be stripped. This used to be the case, but changed in one of these commits: 63305d00d32e8c743e36155a7d8cd544dc676d5b, 443dfc4b05360794aa0304459177557107274559, 2affd455a40a28825f356307ce5bd8fa2f202217

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

Notify maintainers

@amjoseph-nixpkgs

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.4.3-arch1-1, Arch Linux, noversion, rolling`
 - multi-user?: `no`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.16.1`
 - nixpkgs: `not found`

(I think I haven't configured my environment properly, had to pass -I to nix-shell. My nixpkgs commit is currently 63305d00d32e8c743e36155a7d8cd544dc676d5b although as above this bug is still present in master.)

ghost commented 1 year ago

The library should be stripped. This used to be the case, but changed in one of these commits: 63305d0, 443dfc4, 2affd45

Prior to those commits, we were shipping a simply broken cross-compiler which would produce glibcs that were unable to find libgcc_s at all. That's why the closures were smaller before those commits -- they were missing essential libraries!

ghost commented 1 year ago

The library should be stripped.

Are you sure about that? The binaries on Hydra all have unstripped libgcc_so.s.1. None of them have any requisites (nix-store -qR).

From a NixOS machine:

/nix/store/cgg24gknm4gvkzhmvrb7f0sjw87495y6-bootstrap-tools/lib/libgcc_s.so.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
/nix/store/dsn6vl7x1hbn1akgpxync19gpx2dzy8w-bootstrap-tools/lib/libgcc_s.so.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
/nix/store/rkxyrpry5lrr5025ikm92s9j7dmnn1nq-xgcc-12.3.0-libgcc/lib/libgcc_s.so.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
/nix/store/hk4753fvc77v58j7sw5s4cssc09kkca1-bootstrap-tools/lib/libgcc_s.so.1: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped
/nix/store/75337xkp60w4y14dixb7g21j1f82v7b5-xgcc-12.3.0-libgcc/lib/libgcc_s.so.1: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped
/nix/store/mdck89nsfisflwjv6xv8ydj7dj0sj2pn-gcc-11.3.0-lib/lib/libgcc_s.so.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
/nix/store/s36ahabi6s6wdxv71yd6ys4vkma1mb74-gcc-12.3.0-libgcc/lib/libgcc_s.so.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
/nix/store/4nlgxhb09sdr51nc9hdm8az5b08vzkgx-glibc-2.35-163/lib/libgcc_s.so.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
/nix/store/npi0z44ym386livv6pnc10a6dlk9ins9-xgcc-12.3.0-libgcc/lib/libgcc_s.so.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
/nix/store/j1lh75a92bvqc6mac446ss5x21lyhv9b-xgcc-12.3.0-libgcc/lib/libgcc_s.so.1: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped

I think the problem here isn't stripping, it's that your outpath has requisites that it shouldn't. This appears to be specific to your setup and/or arm7l-hf. I'm building that now, but it'll take a while to catch up.

jonathan-conder commented 1 year ago

Prior to those commits, we were shipping a simply broken cross-compiler which would produce glibcs that were unable to find libgcc_s at all. That's why the closures were smaller before those commits -- they were missing essential libraries!

Wasn't aware of that! To be honest I haven't actually tried running anything I've cross-compiled yet. I'm happy to pull in libgcc but I'm pretty sure glibc.dev isn't needed at runtime.

Are you sure about that? The binaries on Hydra all have unstripped libgcc_so.s.1. None of them have any requisites (nix-store -qR).

Yeah, file says not stripped because not everything has been stripped, only the debug symbols. Note that yours don't say with debug_info but the ARM one does.

I think the problem here isn't stripping, it's that your outpath has requisites that it shouldn't.

The only place glibc.dev appears in the outpath is in the debug section of libgcc.

jonathan-conder commented 1 year ago

I'm now thinking this might just be a symptom of another issue. EDIT: nope, this was dumb, I was comparing a cross compiler with a native ARM compiler.

For example:

$ nix develop -i nixpkgs/45ae0efbbce2aada6d5e8de6ace0c803b08ac9c7#pkgsCross.armv7l-hf-multiplatform.hello
bash-5.2$ command -v $CC
/nix/store/n2yr445hpc1pmdngfzz3c45qhhqxmpfi-armv7l-unknown-linux-gnueabihf-stage-final-gcc-wrapper-12.3.0/bin/armv7l-unknown-linux-gnueabihf-gcc
bash-5.2$ exit

$ nix derivation show /nix/store/n2yr445hpc1pmdngfzz3c45qhhqxmpfi-armv7l-unknown-linux-gnueabihf-stage-final-gcc-wrapper-12.3.0 | jq -r 'keys_unsorted[0]'
/nix/store/x9qp4qs32afjkzisk7qrwccncr9458nd-armv7l-unknown-linux-gnueabihf-stage-final-gcc-wrapper-12.3.0.drv
$ nix derivation show nixpkgs/45ae0efbbce2aada6d5e8de6ace0c803b08ac9c7#pkgsCross.armv7l-hf-multiplatform.gcc | jq -r 'keys_unsorted[0]'
/nix/store/3lr834wkr3mw2zshs9d03f7vcq91ysqk-gcc-wrapper-12.3.0.drv

It looks to me like the compiler used by nix develop is further down the bootstrapping chain than the final one:

$ nix derivation show /nix/store/3lr834wkr3mw2zshs9d03f7vcq91ysqk-gcc-wrapper-12.3.0.drv | jq '.[].inputDrvs' | grep gcc
  "/nix/store/n3vznslgl5dbin9m627mh3672jvxgk6s-gcc-armv7l-unknown-linux-gnueabihf-12.3.0.drv": [
$ nix derivation show /nix/store/n3vznslgl5dbin9m627mh3672jvxgk6s-gcc-armv7l-unknown-linux-gnueabihf-12.3.0.drv | jq '.[].inputDrvs' | grep x9
  "/nix/store/x9qp4qs32afjkzisk7qrwccncr9458nd-armv7l-unknown-linux-gnueabihf-stage-final-gcc-wrapper-12.3.0.drv": [

It's probably OK that this compiler has some extra runtime deps as long as the final one doesn't. Not sure why nix develop is picking it though.

jonathan-conder commented 1 year ago

I still think there's something funny going on though. For example:

$ nix path-info -r nixpkgs/45ae0efbbce2aada6d5e8de6ace0c803b08ac9c7#pkgsCross.armv7l-hf-multiplatform.hello | grep glibc
/nix/store/ljc241anx1mis0kygjv40hixbm94phh3-glibc-armv7l-unknown-linux-gnueabihf-2.37-8
$ nix eval --raw nixpkgs/45ae0efbbce2aada6d5e8de6ace0c803b08ac9c7#pkgsCross.armv7l-hf-multiplatform.glibc.out.outPath
/nix/store/v3f8880nzxxv951biarvzv5sf2qz6sdr-glibc-armv7l-unknown-linux-gnueabihf-2.37-8

These 2 are very similar. Apart from dependencies having different hashes, the former depends on stage-static-gcc while the latter depends on stage-final-gcc. Also the latter has some extra code in preConfigure:

makeFlags="$makeFlags BUILD_LDFLAGS=-Wl,-rpath,/nix/store/ljc241anx1mis0kygjv40hixbm94phh3-glibc-armv7l-unknown-linux-gnueabihf-2.37-8/lib OBJDUMP=/nix/store/wz2gx2hvc4jrp09d27qga569i1xk2kbg-armv7l-unknown-linux-gnueabihf-binutils-2.40/bin/objdump"

To me this really does look like the next step in the bootstrap

jonathan-conder commented 1 year ago

Never mind, I think that's just because glibc and glibcCross are different, which I think is intentional?

There's still an issue though: some packages depend on 2 different versions of libgcc. I've been trying out aarch64 just to confirm it's not specific to armv7:

$ git checkout 45ae0efbbce2aada6d5e8de6ace0c803b08ac9c7
$ nix path-info -r .#pkgsCross.aarch64-multiplatform.gmp | grep libgcc
/nix/store/ifwgjiqa6rwdw8r1y4qimmc53p79709h-aarch64-unknown-linux-gnu-stage-static-gcc-12.3.0-libgcc
/nix/store/1h43z24ir2vra4gw86w0xjvnn2p0rx5x-aarch64-unknown-linux-gnu-stage-final-gcc-12.3.0-libgcc

Targets built with pkgsCross.*.stdenv.cc.cc will depend on pkgsCross.*.glibcCross, which depends on pkgsCross.*.glibcCross.stdenv.cc.cc.libgcc:

$ nix why-depends --precise .#pkgsCross.aarch64-multiplatform.gmp .#pkgsCross.aarch64-multiplatform.glibcCross.stdenv.cc.cc.libgcc
/nix/store/gdvhfnbc026pg7dfqglyx5w3w7gvckwp-gmp-with-cxx-aarch64-unknown-linux-gnu-6.2.1
    → /nix/store/zpg16bg1f84p0gydkkbj0qi9nzy6vcj3-glibc-aarch64-unknown-linux-gnu-2.37-8
        → /nix/store/ifwgjiqa6rwdw8r1y4qimmc53p79709h-aarch64-unknown-linux-gnu-stage-static-gcc-12.3.0-libgcc
└───lib/libgmp.so.10.4.1: …gmp.so.10.GLIBC_2.17./nix/store/zpg16bg1f84p0gydkkbj0qi9nzy6vcj3-glibc-aarch64-unknown-linux-gnu…
    └───lib/ld-linux-aarch64.so.1: …inux-gnu-2.37-8/lib/./nix/store/ifwgjiqa6rwdw8r1y4qimmc53p79709h-aarch64-unknown-linux-gnu-stage…

This appears to be due to user-defined-trusted-dirs - maybe glibcCross.stdenv.cc.cc.libgcc should be undefined?

On the other hand, if the target has a direct dependency on libgcc_s, it will use the one in pkgsCross.*.stdenv.cc.cc.libgcc:

$ nix why-depends --precise .#pkgsCross.aarch64-multiplatform.gmp .#pkgsCross.aarch64-multiplatform.stdenv.cc.cc.libgcc
/nix/store/gdvhfnbc026pg7dfqglyx5w3w7gvckwp-gmp-with-cxx-aarch64-unknown-linux-gnu-6.2.1
    → /nix/store/l0sdij8qgssss4pdqbqmqp8ja0fa90ql-aarch64-unknown-linux-gnu-stage-final-gcc-12.3.0-lib
        → /nix/store/1h43z24ir2vra4gw86w0xjvnn2p0rx5x-aarch64-unknown-linux-gnu-stage-final-gcc-12.3.0-libgcc
└───lib/libgmpxx.so.4.6.1: …-linux-gnu-6.2.1/lib:/nix/store/l0sdij8qgssss4pdqbqmqp8ja0fa90ql-aarch64-unknown-linux-gnu-stage…
    └───aarch64-unknown-linux-gnu/lib/libgcc_s.so -> /nix/store/1h43z24ir2vra4gw86w0xjvnn2p0rx5x-aarch64-unknown-linux-gnu-stage-final-gcc-12.3.0-libgcc/lib/libgcc_s.so

I think this is separate from the stripping issue though, since pkgsCross.*.glibcCross.stdenv.cc.cc.libgcc doesn't have unneeded dependencies. If you agree it's a bug I can make a new issue for this.