Enable debug symbols for all packages

[stapelberg]

PR https://github.com/NixOS/nixpkgs/pull/15539 mentions this, but I think it’s worthwhile to have a separate GitHub issue.

Currently, a large number of libraries are lacking debug symbols even though environment.enableDebugInfo = true; is specified in my /etc/nixos/configuration.nix. The following list is taken from a gdb session on NetworkManager:

(gdb) info sharedlibrary 
From                To                  Syms Read   Shared Object Library
0x00007ffff7ddaab0  0x00007ffff7df5990  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/ld-linux-x86-64.so.2
0x00007ffff7bd6630  0x00007ffff7bd7d01  Yes (*)     /nix/store/33z8lb3yzh18z10aqszc07imwlfkyhgb-util-linux-2.28.1/lib/libuuid.so.1
0x00007ffff78c7c80  0x00007ffff7993a44  Yes (*)     /nix/store/3yvgy2mnsm2cfvbjkdcaxlzg1xwh80fj-gnutls-3.4.14/lib/libgnutls.so.30
0x00007ffff764b830  0x00007ffff767a1cf  Yes (*)     /nix/store/yzb1maz797c80606aqj78hafbc4lidcr-p11-kit-0.23.2/lib/libp11-kit.so.0
0x00007ffff7408fd0  0x00007ffff740d6e6  Yes (*)     /nix/store/4b33d5w92qxaw464kgyfs6ncskwmp056-libidn-1.33/lib/libidn.so.11
0x00007ffff71f5d50  0x00007ffff7200b6b  Yes (*)     /nix/store/13lpz809nmpzl12hmn09xnd6yk2jvy7s-libtasn1-4.8/lib/libtasn1.so.6
0x00007ffff6fc4ee0  0x00007ffff6fe15ef  Yes (*)     /nix/store/5rqh0fg8004z9gkwvywbpg75ydf3w9gy-nettle-3.1.1/lib/libnettle.so.6
0x00007ffff6d8f270  0x00007ffff6d9c1e8  Yes (*)     /nix/store/5rqh0fg8004z9gkwvywbpg75ydf3w9gy-nettle-3.1.1/lib/libhogweed.so.4
0x00007ffff6b04280  0x00007ffff6b6ee68  Yes (*)     /nix/store/gb81bp3xbwrhkqd8cw7cvwxl0qlzah4x-gmp-6.1.1/lib/libgmp.so.10
0x00007ffff68edb90  0x00007ffff68f0c62  Yes (*)     /nix/store/z9nhrhcgiqcag9frhdz8lwriypvl8kc5-libgudev-230/lib/libgudev-1.0.so.0
0x00007ffff7fd5c60  0x00007ffff7fe6886  Yes (*)     /nix/store/lamh95q185ymigm217s64x08qcmnfhvm-systemd-231/lib/libudev.so.1
0x00007ffff66d13c0  0x00007ffff66deb35  Yes (*)     /nix/store/vw0wcq2zz1w3l86ly5pzw2dphfpychym-libnl-3.2.28/lib/libnl-3.so.200
0x00007ffff7f59760  0x00007ffff7fae2aa  Yes (*)     /nix/store/lamh95q185ymigm217s64x08qcmnfhvm-systemd-231/lib/libsystemd.so.0
0x00007ffff64c4fe0  0x00007ffff64c6784  Yes (*)     /nix/store/z4mg6wiy5y26p0djr95b4b6cnpf3m7qb-libndp-1.6/lib/libndp.so.0
0x00007ffff622a2e0  0x00007ffff6270aef  Yes (*)     /nix/store/3dg2ld3q204xsi2ml12073nw9bng7xf7-libsoup-2.54.1/lib/libsoup-2.4.so.1
0x00007ffff5eb07a0  0x00007ffff5f93160  Yes (*)     /nix/store/9didfvj6wba723zcnls1c5iwp9jv3l3l-libxml2-2.9.4/lib/libxml2.so.2
0x00007ffff5b83660  0x00007ffff5beffba  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libm.so.6
0x00007ffff58b2440  0x00007ffff5955100  Yes (*)     /nix/store/1bx29pqdzwc722an88nzxxvn24al2nyb-sqlite-3.14.1/lib/libsqlite3.so.0
0x00007ffff5555490  0x00007ffff562892b  Yes (*)     /nix/store/nvclglzzqj57gzclmq44aziw33y7d2c4-glib-2.48.2/lib/libgio-2.0.so.0
0x00007ffff531d2b0  0x00007ffff531e1b5  Yes (*)     /nix/store/nvclglzzqj57gzclmq44aziw33y7d2c4-glib-2.48.2/lib/libgmodule-2.0.so.0
0x00007ffff51081a0  0x00007ffff51143d9  Yes (*)     /nix/store/ahjpbwi957n7pqp7i7ys6s2vxmk4dr06-zlib-1.2.8/lib/libz.so.1
0x00007ffff4ef2810  0x00007ffff4efe93b  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libresolv.so.2
0x00007ffff4ca79e0  0x00007ffff4cd83a6  Yes (*)     /nix/store/nvclglzzqj57gzclmq44aziw33y7d2c4-glib-2.48.2/lib/libgobject-2.0.so.0
0x00007ffff4a95aa0  0x00007ffff4a9a35c  Yes (*)     /nix/store/3zzw0gc6078nsp829x2va9690kca237j-libffi-3.2.1/lib/../lib64/libffi.so.6
0x00007ffff479ef80  0x00007ffff4815bb9  Yes (*)     /nix/store/nvclglzzqj57gzclmq44aziw33y7d2c4-glib-2.48.2/lib/libglib-2.0.so.0
0x00007ffff4516730  0x00007ffff4565996  Yes (*)     /nix/store/10jk5h18ahh2xaad74ng3mgx0lfjmfj6-pcre-8.38/lib/libpcre.so.1
0x00007ffff42fd810  0x00007ffff430a531  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libpthread.so.0
0x00007ffff40f4d70  0x00007ffff40f590e  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libdl.so.2
0x00007ffff3d75930  0x00007ffff3e9e1a3  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libc.so.6
0x00007ffff3b526d0  0x00007ffff3b53fd6  Yes (*)     /nix/store/4am908pjq2j4bzqcy051mx62ab0rd1xa-libcap-2.25-lib/lib/libcap.so.2
0x00007ffff394af70  0x00007ffff394dd1f  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/librt.so.1
0x00007ffff3725e30  0x00007ffff373bff2  Yes (*)     /nix/store/3qq5qv97jdqg0ywq14spj7kmif39fhnw-xz-5.2.2/lib/liblzma.so.5
0x00007ffff35132a0  0x00007ffff351f5bb  Yes (*)     /nix/store/bppnjqhm8jjbwc8215i9qvg4ws39avz1-lz4-131/lib/liblz4.so.1
0x00007ffff320ce00  0x00007ffff32cce48  Yes (*)     /nix/store/i7z7i541bfa6gxw1g71wzf82gj5ms8s7-libgcrypt-1.7.3/lib/libgcrypt.so.20
0x00007ffff2ff0a30  0x00007ffff2ffa099  Yes (*)     /nix/store/4ixpz8hqaqv4h4hibm0m8ljpbv38wd3n-libgpg-error-1.24/lib/libgpg-error.so.0
0x00007fffea9c30d0  0x00007fffea9c8d91  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libnss_files.so.2
0x00007fffea7a6800  0x00007fffea7b74c6  Yes         /tmp/nm/nix/store/g05c3qil03b5lg02nqlz5viv2ad3bsam-network-manager-1.2.2/lib/NetworkManager/libnm-device-plugin-wifi.so
(*): Shared library is missing debugging information.

In the above list, only glibc seems to come with debug symbols, no other library does.

https://nixos.org/nixos/manual/options.html#opt-environment.enableDebugInfo explains how to override individual packages, but that’s tedious and requires recompilation of all reverse-dependencies, which is really really inconvenient (my machine is compiling spidermonkey since half an hour… :-/).

Debian recently started building debug symbol paackages by default and Fedora seems to have debuginfo RPMs built by default as well. I think having debug symbols on by default is a feature that users of modern Linux distributions have come to expect :).

Could we enable debug symbols for all packages by default please?

cc @kevincox

[lheckemann]

That's a very good point. I think (hope) that it would be rare enough for debug symbols to be incompatible with the binaries that this isn't a major issue, but if it ever does happen we need a way to get a derivation rebuilt with the outputs replacing the previous versions... and I'm not sure what the best ergonomics for that would be. I have three ideas currently:

• Alter nix-store --repair-path so that it will rebuild the path even if it's not corrupted; this would allow solving the problem with nix-store --repair-path $path --substituters ''. This is the closest we already have to this functionality, AFAIK;
• A new flag for nix-build and nix-store -r which would behave sort of like --check (in that it rebuilds the path even if it's already there or if it can be substituted) and sort of like --repair-path (replacing what's currently in the store);
• A new command, nix rebuild.

I'm not very happy with any of these, if anyone has any good ideas I'd appreciate them!

[Gaelan]

There's a simpler solution here (I think): modify Nix to never include two outputs from different sources in the store. So if you have foo-bin in the store, and you ask for foo-debug, Nix either:

• gets foo-debug from the same substituter that it got foo-bin from (if this is a thing Nix knows); if we don't have this information, or if that substituter doesn't have foo-debug, we:
• look for another substituter with both foo-bin and foo-debug, and download that; and if no such substituter exists, then:
• build foo ourselves and put all the outputs into the store

[lheckemann]

Of course! While tracking provenance of paths is complicated (Nix doesn't currently do this), replacing all outputs with what's been built as in your last suggestion seems pretty feasible to me. Maybe Nix even already does this. Thanks :D

[lheckemann]

Maybe Nix even already does this.

Unfortunately, it doesn't:

Script I used to test this

```bash #!/usr/bin/env bash set -exuo pipefail export NIX_PATH=nixpkgs=/home/linus/nixpkgs/master root=/scratch/missing-output-test cachestore=$root/cache deststore=$root/testing for store in $deststore; do if [[ -d "$store" ]]; then chmod -R u+w $store rm -r $store fi mkdir -p $store done opts=(--secret-key-files '' --builders '') cd $root cat >expr.nix <<\EOF let pkgs = import {}; in rec { hello = pkgs.hello.overrideAttrs (o: { preConfigure = "exec 5>&2 2>/dev/null >/dev/null"; separateDebugInfo = true; postFixup = "date | tee $out/date $debug/date"; }); refhello = pkgs.runCommand "refhello" {} '' date > $out echo ${hello} >> $out ''; } EOF drv=$(nix-instantiate "${opts[@]}" --store $cachestore expr.nix -A refhello) out=$(nix-instantiate "${opts[@]}" --store $cachestore --eval expr.nix -A hello.out.outPath --json | jq -r) debug=$(nix-instantiate "${opts[@]}" --store $cachestore --eval expr.nix -A hello.debug.outPath --json | jq -r) ref=$(nix-instantiate "${opts[@]}" --store $cachestore --eval expr.nix -A refhello.outPath --json | jq -r) nix-store "${opts[@]}" --store $cachestore -r $drv nix-store "${opts[@]}" --store $cachestore --delete $debug nix-build "${opts[@]}" --store $deststore --extra-substituters $cachestore --no-require-sigs expr.nix -A refhello nix-build "${opts[@]}" --store $deststore --extra-substituters $cachestore --no-require-sigs expr.nix -A hello.debug grep . {$cachestore,$deststore}/{{$out,$debug}/date,$ref} ```

I suspect it shouldn't be too hard to change that though.

[layus]

There is no solution for that, it is a consequence of nix design. Replacing all the outputs just defers the issue to packages down the dependency chain. Also, as soon as you have multiple caches, or a cache starts dropping entries, you get this issue. Replacing all the outputs fixes inter-outputs provenance incompatibilities. But then all the packages that depended on the previous, replaced output may suffer from inconsistencies with the new one. Even though there should be even less inconsistencies between packages than between outputs of a single package.

I just wanted to make a note about this blind spot in the design of nix, that remains well hidden as long as we all rely on a shared central cache that does not discard any path. And discarding some paths is exactly the solution to the problem at hand here.

So let's leave a note here about this issues for people to find out if they ever encounter this, and let's move on.

Who has enough accesses to enable to make the required tests and changes ?

[edolstra]

modify Nix to never include two outputs from different sources in the store.

This isn't going to happen because it's a pretty fundamental assumption in Nix that store paths with the same name are substitutable for each other (whether by actually substituting them from a binary cache, or by (re)building them locally). But this issue is moot as long as builds are binary-reproducible, which is what we should strive for anyway. (E.g. CAS Nix doesn't work as well if packages are not binary-reproducible.)

[lheckemann]

Who has enough accesses to enable to make the required tests and changes ?

I suppose the next step would be hydra support for selectively copying outputs to the cache. Anyone could implement this :)

[Ericson2314]

@layus I think with CA derivations this would be solved, because the debug output references the original, it will require it match a content-address; mixing output from non-deterministic builds won't work.

[layus]

@Ericson2314 Yes indeed, because CA preserves the provenance relation ;-).

[Ericson2314]

(C.F. #4344)

[vcunat]

Debug info size: as we're about to default to gcc12 soon, it might be interesting to look into -gctf1 which should be way smaller than normal debug info (-g / -ggdb) and still provide backtraces. https://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html

The format is just from the past few years though, so it's likely there will be some hurdles to solve first.

[symphorien]

We should however keep in mind that it is not perfect. There is a risk of franken(debug)builds, namely the possibility that you associate a locally re-generated *-debug output with a cached output, and get mismatches because of build impurities. This would not happen if all the builds were perfectly reproducible, but that is not the case. See https://gist.github.com/layus/2a91694be330d5f2591dab9a52c47548 for a mwe.

this is already happening for qemu https://github.com/NixOS/nix/issues/7756

[symphorien]

https://github.com/NixOS/nix/pull/8080 modifies nix so that it tags debug outputs when uploading them to s3 in a way that makes it possible to configure the s3 bucket backing cache.nixos.org to automatically remove debug outputs and debug outputs only after a configurable period of time.