Enable debug symbols for all packages

stapelberg commented 8 years ago

PR https://github.com/NixOS/nixpkgs/pull/15539 mentions this, but I think it’s worthwhile to have a separate GitHub issue.

Currently, a large number of libraries are lacking debug symbols even though environment.enableDebugInfo = true; is specified in my /etc/nixos/configuration.nix. The following list is taken from a gdb session on NetworkManager:

(gdb) info sharedlibrary 
From                To                  Syms Read   Shared Object Library
0x00007ffff7ddaab0  0x00007ffff7df5990  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/ld-linux-x86-64.so.2
0x00007ffff7bd6630  0x00007ffff7bd7d01  Yes (*)     /nix/store/33z8lb3yzh18z10aqszc07imwlfkyhgb-util-linux-2.28.1/lib/libuuid.so.1
0x00007ffff78c7c80  0x00007ffff7993a44  Yes (*)     /nix/store/3yvgy2mnsm2cfvbjkdcaxlzg1xwh80fj-gnutls-3.4.14/lib/libgnutls.so.30
0x00007ffff764b830  0x00007ffff767a1cf  Yes (*)     /nix/store/yzb1maz797c80606aqj78hafbc4lidcr-p11-kit-0.23.2/lib/libp11-kit.so.0
0x00007ffff7408fd0  0x00007ffff740d6e6  Yes (*)     /nix/store/4b33d5w92qxaw464kgyfs6ncskwmp056-libidn-1.33/lib/libidn.so.11
0x00007ffff71f5d50  0x00007ffff7200b6b  Yes (*)     /nix/store/13lpz809nmpzl12hmn09xnd6yk2jvy7s-libtasn1-4.8/lib/libtasn1.so.6
0x00007ffff6fc4ee0  0x00007ffff6fe15ef  Yes (*)     /nix/store/5rqh0fg8004z9gkwvywbpg75ydf3w9gy-nettle-3.1.1/lib/libnettle.so.6
0x00007ffff6d8f270  0x00007ffff6d9c1e8  Yes (*)     /nix/store/5rqh0fg8004z9gkwvywbpg75ydf3w9gy-nettle-3.1.1/lib/libhogweed.so.4
0x00007ffff6b04280  0x00007ffff6b6ee68  Yes (*)     /nix/store/gb81bp3xbwrhkqd8cw7cvwxl0qlzah4x-gmp-6.1.1/lib/libgmp.so.10
0x00007ffff68edb90  0x00007ffff68f0c62  Yes (*)     /nix/store/z9nhrhcgiqcag9frhdz8lwriypvl8kc5-libgudev-230/lib/libgudev-1.0.so.0
0x00007ffff7fd5c60  0x00007ffff7fe6886  Yes (*)     /nix/store/lamh95q185ymigm217s64x08qcmnfhvm-systemd-231/lib/libudev.so.1
0x00007ffff66d13c0  0x00007ffff66deb35  Yes (*)     /nix/store/vw0wcq2zz1w3l86ly5pzw2dphfpychym-libnl-3.2.28/lib/libnl-3.so.200
0x00007ffff7f59760  0x00007ffff7fae2aa  Yes (*)     /nix/store/lamh95q185ymigm217s64x08qcmnfhvm-systemd-231/lib/libsystemd.so.0
0x00007ffff64c4fe0  0x00007ffff64c6784  Yes (*)     /nix/store/z4mg6wiy5y26p0djr95b4b6cnpf3m7qb-libndp-1.6/lib/libndp.so.0
0x00007ffff622a2e0  0x00007ffff6270aef  Yes (*)     /nix/store/3dg2ld3q204xsi2ml12073nw9bng7xf7-libsoup-2.54.1/lib/libsoup-2.4.so.1
0x00007ffff5eb07a0  0x00007ffff5f93160  Yes (*)     /nix/store/9didfvj6wba723zcnls1c5iwp9jv3l3l-libxml2-2.9.4/lib/libxml2.so.2
0x00007ffff5b83660  0x00007ffff5beffba  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libm.so.6
0x00007ffff58b2440  0x00007ffff5955100  Yes (*)     /nix/store/1bx29pqdzwc722an88nzxxvn24al2nyb-sqlite-3.14.1/lib/libsqlite3.so.0
0x00007ffff5555490  0x00007ffff562892b  Yes (*)     /nix/store/nvclglzzqj57gzclmq44aziw33y7d2c4-glib-2.48.2/lib/libgio-2.0.so.0
0x00007ffff531d2b0  0x00007ffff531e1b5  Yes (*)     /nix/store/nvclglzzqj57gzclmq44aziw33y7d2c4-glib-2.48.2/lib/libgmodule-2.0.so.0
0x00007ffff51081a0  0x00007ffff51143d9  Yes (*)     /nix/store/ahjpbwi957n7pqp7i7ys6s2vxmk4dr06-zlib-1.2.8/lib/libz.so.1
0x00007ffff4ef2810  0x00007ffff4efe93b  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libresolv.so.2
0x00007ffff4ca79e0  0x00007ffff4cd83a6  Yes (*)     /nix/store/nvclglzzqj57gzclmq44aziw33y7d2c4-glib-2.48.2/lib/libgobject-2.0.so.0
0x00007ffff4a95aa0  0x00007ffff4a9a35c  Yes (*)     /nix/store/3zzw0gc6078nsp829x2va9690kca237j-libffi-3.2.1/lib/../lib64/libffi.so.6
0x00007ffff479ef80  0x00007ffff4815bb9  Yes (*)     /nix/store/nvclglzzqj57gzclmq44aziw33y7d2c4-glib-2.48.2/lib/libglib-2.0.so.0
0x00007ffff4516730  0x00007ffff4565996  Yes (*)     /nix/store/10jk5h18ahh2xaad74ng3mgx0lfjmfj6-pcre-8.38/lib/libpcre.so.1
0x00007ffff42fd810  0x00007ffff430a531  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libpthread.so.0
0x00007ffff40f4d70  0x00007ffff40f590e  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libdl.so.2
0x00007ffff3d75930  0x00007ffff3e9e1a3  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libc.so.6
0x00007ffff3b526d0  0x00007ffff3b53fd6  Yes (*)     /nix/store/4am908pjq2j4bzqcy051mx62ab0rd1xa-libcap-2.25-lib/lib/libcap.so.2
0x00007ffff394af70  0x00007ffff394dd1f  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/librt.so.1
0x00007ffff3725e30  0x00007ffff373bff2  Yes (*)     /nix/store/3qq5qv97jdqg0ywq14spj7kmif39fhnw-xz-5.2.2/lib/liblzma.so.5
0x00007ffff35132a0  0x00007ffff351f5bb  Yes (*)     /nix/store/bppnjqhm8jjbwc8215i9qvg4ws39avz1-lz4-131/lib/liblz4.so.1
0x00007ffff320ce00  0x00007ffff32cce48  Yes (*)     /nix/store/i7z7i541bfa6gxw1g71wzf82gj5ms8s7-libgcrypt-1.7.3/lib/libgcrypt.so.20
0x00007ffff2ff0a30  0x00007ffff2ffa099  Yes (*)     /nix/store/4ixpz8hqaqv4h4hibm0m8ljpbv38wd3n-libgpg-error-1.24/lib/libgpg-error.so.0
0x00007fffea9c30d0  0x00007fffea9c8d91  Yes         /nix/store/y5js9j7zgrmz4j2dqcy0nvaqyk3izbv7-glibc-2.24/lib/libnss_files.so.2
0x00007fffea7a6800  0x00007fffea7b74c6  Yes         /tmp/nm/nix/store/g05c3qil03b5lg02nqlz5viv2ad3bsam-network-manager-1.2.2/lib/NetworkManager/libnm-device-plugin-wifi.so
(*): Shared library is missing debugging information.

In the above list, only glibc seems to come with debug symbols, no other library does.

https://nixos.org/nixos/manual/options.html#opt-environment.enableDebugInfo explains how to override individual packages, but that’s tedious and requires recompilation of all reverse-dependencies, which is really really inconvenient (my machine is compiling spidermonkey since half an hour… :-/).

Debian recently started building debug symbol paackages by default and Fedora seems to have debuginfo RPMs built by default as well. I think having debug symbols on by default is a feature that users of modern Linux distributions have come to expect :).

Could we enable debug symbols for all packages by default please?

cc @kevincox

edolstra commented 8 years ago

I don't know if we should enable it for all packages. We should probably have some guesstimate of the additional storage costs. However, it's definitely a good idea to enable it for some major libraries.

stapelberg commented 8 years ago

I’m assuming you’re referring to the build output stored on build servers, since the debug info on the disks of end users is configurable using enableDebugInfo :).

For comparison, here’s a list of the sizes (in bytes) of all 1726 Debian packages with a “-dbg” suffix currently in Debian testing (amd64): debian-debug-sizes.txt The sum of all of these packages is about 10 GiB.

Here’s a list of the sizes (in percent, e.g. 100 is 100%) relative to the corresponding package: relative.txt The average size of a debug package is 4557% of the base package.

Note that these should be treated as rough estimates: the base package matching was done textually (as opposed to understanding the source-package relationship).

Based on these values from Debian, can you do a guesstimate as to how much additional space would be required? AFAICT (without any knowledge of the NixOS build infrastructure), the additional demand seems to range in the tens of gigabytes region, which should be totally doable…?

kevincox commented 8 years ago

I would love to see this happen. As for my other PR life got in the way and there was debate that never got resolved. The basics work but you can see the checkboxes to see what features I would like added (although they can likely be added as a later patch). I should have time to pick it up around now, hopefully banging it into a shape that is generally agreed to be acceptable.

As for the package sizes this is obviously a code tradeoff and it will depend on what hydra/S3 can have available. It would definitely be awesome to have most libraries built with debug symbols.

copumpkin commented 8 years ago

Would we put them in a separate output?

kevincox commented 8 years ago

On Sep 12, 2016 23:08, "Daniel Peebles" notifications@github.com wrote:

Would we put them in a separate output?

I'm pretty sure this is a requirement.

vcunat commented 8 years ago

Would we put them in a separate output?

It should all be just a matter of adding separateDebugInfo = true; into the particular packages.

Note: the debian list you provided has 1726 lines. One Hydra evaluation of ours contains way more builds (~35k). There are some duplicates in there, and many would only have negligible debug info, but there still might be a significant difference. I can't see how to find easily, except to build one generation in that way.

stapelberg commented 8 years ago

I’m aware of that. Debian has >20k source packages, so we’re talking about a roughly 10% sample size here at best — that should be representative enough for a guesstimate.

I agree that the easiest way to get accurate numbers would be to just build a generation with debug symbols enabled, though :).

vcunat commented 8 years ago

What's their total size of non-debug stuff on this list? (EDIT: ratio of sums or averages seems much more reliable than average of ratios.)

stapelberg commented 8 years ago

It’s 8.19G of debug packages and 0.97G of corresponding non-debug packages.

vcunat commented 8 years ago

Well, 9-fold increase of required space would be a disaster. Hopefully they don't compress their debug files; we do it.

stapelberg commented 8 years ago

On the package level, compression is not accounted for in my numbers: the sizes are for the unpacked, uncompressed package contents.

On the debug file level, I can’t tell whether they are compressed or not…?

$ file /usr/lib/debug/.build-id/ee/13cb8a18d72bea00fd7f7ff22544d3024eb5c3.debug
/usr/lib/debug/.build-id/ee/13cb8a18d72bea00fd7f7ff22544d3024eb5c3.debug: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter *empty*, for GNU/Linux 2.6.32, BuildID[sha1]=ee13cb8a18d72bea00fd7f7ff22544d3024eb5c3, not stripped

vcunat commented 8 years ago

Hmm, I see the compression doesn't make that great difference (~30 %) and even after it the debug files are expected to be 2–4x size of the corresponding binaries (for complex C++ at least). https://gcc.gnu.org/wiki/DebugFission But we store more than just ELFs.

We've got glibc example where the increase is ~40 %:

22M     /nix/store/6fix3zqpnahyml8zp2sxi2rwan55rgb8-glibc-2.24
2.7M    /nix/store/g2f08f2gb4dpc2l0g57jy6yp26bfc4hj-glibc-2.24-bin
14M     /nix/store/2gfgppppk96xi91xv90xlz85z73m8prj-glibc-2.24-debug
3.2M    /nix/store/8sl4jfs3nq0pkq4gg655s3axrxdx7z29-glibc-2.24-dev
8.2M    /nix/store/m2mvna9pk9hv1qgxwi124avzdq1i5q96-glibc-2.24-static
50M     total

Perhaps if we restricted the system-wide enabling to x86_64-linux for now, the total increase might be as small as 10 %.

vandenoever commented 8 years ago

Having debug symbols enabled should be the default in my opinion, especially in a partially self-compiled distributions like NixOS. The threshold to submitting bugs should be low. Great to see it as a milestone for 17.03.

0xABAB commented 7 years ago

Storage costs also depend on the device being used to store them on. On some systems, I would much rather store the debugging symbols on cheap hard disk space as opposed to an NVME or SSD disk, while keeping the rest of the Nix store on fast storage.

For me it would be the difference between enabling debug symbols for everything and not.

kevincox commented 7 years ago

@0xABAB I suspect that you wouldn't install debugging symbols on most machines, they would be kept on your binary cache until you need to symbolize a stack trace. This would likely be done on developer machines or dedicated servers.

0xABAB commented 7 years ago

@kevincox In that case the specific feature being discussed here is not clear.

As a developer oriented user, I would like to:

run a program with a segfault
get a dialog that it has crashed
have an option in configuration.nix to say to automatically download debugging symbols on demand
see a beautiful back trace which in KDE's user interface has three stars or contains no question marks/hidden symbols in simple English.
purge these downloaded debugging symbols automatically after two weeks if free disk space is below 30%.

I can imagine an operator of a binary cache to want low operational costs, but it seems that with a properly designed Nix the costs are only dependent on the costs of S3, not of the server's attached storage.

I don't see why any user would ever be required to build packages. @volth apparently has reasons to believe there are still cases like that, but I don't see them.

domenkozar commented 7 years ago

I think the next step is rebuilding nixos-small with debug on and compressing all debug outputs to xz to see how much each package size increases on average.

edolstra commented 7 years ago

.debug files are already compressed using --compress-debug-sections=zlib. (Whether this is useful is questionable, given that NARs in the binary cache are already compressed using xz, so probably we just end up with a worse compression ratio...)

Note that since a few months, cache.nixos.org provides a index of .debug files (indexed by their build ID). For example, to get the debug symbols for libc.so.6:

$ readelf -a /nix/store/kjwbqnh13dxh6w4pk2gb3ddmhpiaihqg-glibc-2.25/lib/libc.so.6 | grep 'Build ID'
    Build ID: 443f38a7db6510d81a7afe03b1a42de8ef6c6ebf

$ curl https://cache.nixos.org/debuginfo/443f38a7db6510d81a7afe03b1a42de8ef6c6ebf
{"archive":"../nar/0xx62iin8dpql0fn70ad7fy8d2w4vzc10d7nbl84al8k723np3p4.nar.xz","member":"lib/debug/.build-id/44/3f38a7db6510d81a7afe03b1a42de8ef6c6ebf.debug"}

So https://cache.nixos.org/nar/0xx62iin8dpql0fn70ad7fy8d2w4vzc10d7nbl84al8k723np3p4.nar.xz contains the desired debug symbols.

I also wrote a little FUSE file system that fetches debug symbols automatically from the binary cache. So you can mount the file system on (say) /run/debug, point the NIX_DEBUG_INFO_DIRS environment variable to it, and gdb and eu-stack will be able to obtain debug info for any pacakge in the binary cache that has debug info enabled.

domenkozar commented 7 years ago

@edolstra if I understand correctly one needs to recompile nixpkgs with separateDebugInfo = true to get this working right?

edolstra commented 7 years ago

@domenkozar Yes.

lheckemann commented 6 years ago

In general, what would the preferred solution for actually getting everything building with debug symbols be, assuming that the storage problem is somehow solved (even if it's just by not uploading debug outputs to the binary cache)?

edolstra commented 6 years ago

I think that would just be a one-line change to stdenv to set separateDebugInfo by default.

lheckemann commented 6 years ago

I tried

diff --git a/pkgs/stdenv/generic/make-derivation.nix b/pkgs/stdenv/generic/make-derivation.nix
index 7b5f9f7d6b0..ab506267c54 100644
--- a/pkgs/stdenv/generic/make-derivation.nix
+++ b/pkgs/stdenv/generic/make-derivation.nix
@@ -59,7 +59,7 @@ rec {
         (if attrs.meta.description or null != null
           then builtins.unsafeGetAttrPos "description" attrs.meta
           else builtins.unsafeGetAttrPos "name" attrs)
-    , separateDebugInfo ? false
+    , separateDebugInfo ? ! (attrs ? outputHashAlgo || attrs ? buildCommand)
     , outputs ? [ "out" ]
     , __impureHostDeps ? []
     , __propagatedImpureHostDeps ? []

Because fixed-output derivations can't have multiple outputs, and because buildCommand overrides fixupPhase, but there's still a bunch of stuff that fails to build (because of not producing the debug output) even inside stdenv.

layus commented 6 years ago

Is there some way to enable separateDebugInfo based on the presence of a c/cpp/... compiler in the build inputs ? That would force to use stdenvNoCC when there is no need to. And would also help catching rogue overrides of buildCommand.

vcunat commented 6 years ago

It's not difficult, but I think it would be over-optimistic. Standard stdenv is still used in many derivations that don't produce (c-style) binaries.

timor commented 5 years ago

Is the motivation for this issue being able to easily run a debugger for a specific package? When I played around with debug information, I noticed that in addition to the debug symbols, the relevant source code is usually also not there anymore.

I handled that problem with this quite intrusive piece of nix code: https://github.com/timor/timor-overlay/blob/master/default.nix#L16-L108, which might be of interest for this issue.

The basic idea is to be able to source-level-debug a certain package and its (manually specified) dependencies. The way it works is that the source code of the specified packages is saved after build, debug symbols generated, and then an environment dedicated for running gdb for that particular set of packages is generated.

That by itself does not provide zero-effort stack traces for arbitrary applications, though. It also could be extended to trickle down the separateDebugInfo attribute to its dependencies, if not already enabled.

matthewbauer commented 5 years ago

We could do mkdir -p $debug in these cases to avoid the failure.

matthewbauer commented 5 years ago

Is everyone on board to try this for 19.09? I think it's a good idea to do this by default. Having a few of these be empty directories doesn't seem too bad.

FWIW I have a branch to get this working with LLVM/Darwin builds here:

https://github.com/matthewbauer/nixpkgs/commits/macho

It's probably better to get this figured out on Linux first, but it seems to be fairly easy to get this working with macho binaries.

edolstra commented 5 years ago

@matthewbauer On board with what exactly? Enabling debug info for all packages? Exporting sources? Adding mkdir -p $debug?

matthewbauer commented 5 years ago

I mean to try to enable all debug symbols globally. I don't really care about specifics, but the mkdir -p $debug hack seemed like the easiest way to avoid tons of separateDebugInfo = false for everything in the tree.

edolstra commented 5 years ago

I'm not sure if that's a good idea in light of some of the statistics above (e.g. https://github.com/NixOS/nixpkgs/issues/18530#issuecomment-246820586). It seems that enabling debug symbols would significantly increase storage costs and increase build times (since debug symbols need to be uploaded to S3).

matthewbauer commented 5 years ago

That seems reasonable if it really is the costly. Perhaps we could try some smaller set of packages like haskellPackages and enable it there? That has been requested a few times here: https://github.com/NixOS/nixpkgs/issues/51987

nh2 commented 5 years ago

enabling debug symbols would significantly increase storage costs and increase build times

I'd like to add that having debug symbols for all packages easily available would be a big boon for troubleshooting issues in production quickly.

Debian/Ubuntu's -dbg packages are super useful here, and NixOS should have them too to be equally useful!

I don't have a good answer to build times, but when it comes to storage, I wouldn't be surprised if the cost of one of us debugging a production segfault without debug info once per year far outweighs the storage costs for debug info.

vcunat commented 5 years ago

Our main stumbling point seems to be the uncertainty about the size this adds, so now I tried to get an estimate of what tree-wide separateDebugInfo = true would do in our NixPkgs.

ATM the sample is smaller than I'd like, but it seems likely that growth of one nixos-unstable evaluation (on x86_64-linux) wouldn't be too far from 20–25 GB. I expect that would be OK-ish, especially if restricted to this single platform (for now?). I'm not sure if a separate GC strategy for these is easy to implement for our S3 (suggested above, good idea).

Notes:

I suspect this flag isn't enough for haskellPackages, as their sizes seem suspiciously tiny. This experiment could be extended... or done specifically on a subset like haskellPackages (greater accuracy with similarly small sample size).
A large part of our packages are various languages (python, perl, ...), which could explain why our ratio of sizes seems very different from Debian.
Sampling like this will always suffer from the risk of missing "black swans"... but we could find such cases retrospectively and just disable .debug for them if they don't seem worth it. They statistically can't be in a very large number – a couple percent (of count) at most, even with this small sample size.
We'll need some tweaks besides just enabling the flag, otherwise quite a large fraction of packages breaks. I did some quick hacks to pull this off and suggested a few more: https://github.com/vcunat/nixpkgs/commits/v/try-debug

Methodology, more numbers

I downloaded a store-path list for a current channel release. The paths are ordered by crypto-hashes, so a simple prefix makes quite a high-quality random sample. I "assigned" the (missing) `.debug` to each `.out` I encountered in that sample, and found its `du`-size when rebuilt. The debug info is compressed, so .nar.xz would be just a few percent smaller (I tried `xz -9` on a couple). Estimate ATM:

88904 KiB * 33667/126  / 1024^2 = ~22.7 GiB

I consider the sample small, because the distribution is clearly *very* asymmetric. For nice one, mere 30 would be fine. Non-zero tail:

8       /nix/store/1ccjbi37bnslppim88kq9xbncf6ny45j-nano-erl-0.1.0.1-debug
8       /nix/store/31dydkpmipn89gz6a5kq9jdjww3iwsw8-prettyprinter-compat-ansi-wl-pprint-1.0.1-debug
8       /nix/store/4rq0vni3v43779sgpmhvkjygswafrmkz-restless-git-0.7-debug
8       /nix/store/6a1nlcj161q90xs146ip9af00bq1xvlr-ifstat-legacy-1.1-debug
8       /nix/store/732yhkf8hq0gm3jpwixi9xiyl5ghs144-text-icu-0.7.0.1-debug
8       /nix/store/85px3zyajm2nxiha5qf7d32b1g6a3ln2-bench-1.0.12-debug
8       /nix/store/8kj422wv2m1c8731762h1ywd2c7qmmlf-timer-wheel-0.1.0-debug
8       /nix/store/98mwjfw4nh3vnqvasq5ji1a7ndl0ixwb-leancheck-instances-0.0.3-debug
8       /nix/store/9cdg2nd5xfjbpx1g08rmhg122kzyxd3l-tmp-postgres-0.1.1.1-debug
8       /nix/store/9gx8yyq5cvi99fk6n5miqjf7457nrd64-monad-hash-0.1.0.2-debug
8       /nix/store/anid0qr48hjy0gk2x4kfbqkj65hr08b4-acl2-0.0.1-debug
8       /nix/store/cy7x1ldfvwpx4dpn667r8v6qypjkjqv7-alarmclock-0.6.0.2-debug
8       /nix/store/igrz4jm4ah8877myay2g8qkv0an4c41i-complex-integrate-1.0.0-debug
8       /nix/store/jrix237vl6navdh8s1y0gkrxrlz7yz39-detrospector-0.3-debug
8       /nix/store/kz06bh1afrja7h0q7anzxy2vph8d8cxn-aeson-generic-compat-0.0.1.3-debug
8       /nix/store/lcfayl3ylz0zq3ap33grhcr2yny6cniv-html2hamlet-0.3.0-debug
8       /nix/store/n7lvvr1cgqsxm7v81n4yl3d6w21qwc8a-language-cil-0.4.0-debug
8       /nix/store/p2x5ccdwyziibncb3ilq81l7y9h39663-osx-ar-0.11-debug
8       /nix/store/qxlrd12yk9q65kmxdhv3c945d4svwfg5-yesod-transloadit-0.7.1.0-debug
8       /nix/store/ry2v88mcwk7zji4kw2q736n3p7pcbya5-wrap-0.0.0-debug
8       /nix/store/s1a0xd0ci1bkvqaqz1vlvw3lwsvwvh7h-multiplicity-0.1.0-debug
8       /nix/store/snlj5p6mkqr64vfirw3cz7i9f0hzv7n0-amazonka-efs-1.6.1-debug
8       /nix/store/wq5fa2hg68rr0c31l6g551bc5z8xrkwz-machines-0.6.4-debug
16      /nix/store/andq2i1shnpn5p2c5xi9c7qbw46xpai6-fst-0.10.0.1-debug
16      /nix/store/n6vpf2ngj0bc3ccicgc42x3wk8cf0pzd-cab-0.2.18-debug
16      /nix/store/sg2p75lf4v54v2h9jacks70zwcg8zdw8-foobar-0.1.0.0-debug
24      /nix/store/dc1qs1hvfgcs84n9jyhbfm1zpv5rnz5p-perfect-hash-generator-0.2.0.6-debug
24      /nix/store/f7rgh0s8py4w04panm9anq7qph6zza5v-msilbc-2.1.2-debug
32      /nix/store/72c6bjyvbgvm1knrpm25cyaxdmynmpxm-elementary-print-shim-0.1.3-debug
32      /nix/store/bbqihknyq85nj163bk6w9cl2bl4ssf9h-lua5.2-lrexlib-posix-2.9.0-1-debug
44      /nix/store/vk4bgrfmxd6hzb2a7vy72rr4547yam3c-smemstat-0.02.00-debug
48      /nix/store/87lqfns07js7bcf1761qd6iy8h8whrq0-ispell-3.3.02-debug
52      /nix/store/wy6x5yhpkj8gi0598h929ajf1sjv8xdf-perl5.29.6-B-Utils-0.27-debug
64      /nix/store/9cs6cvvpxp75vfl24fhk7a3r1y58bsbp-python2.7-simplejson-3.16.0-debug
72      /nix/store/86gxba59x9xfjzh3797m01mdwcfih3mg-armadillo-9.200.7-debug
112     /nix/store/8xnnq1h33zc3xyz84hisgc28l25qxakc-gpsd-3.16-debug
116     /nix/store/xskbklx0isbfyk8qgx58wl4c3pdp0ppx-libmilter-8.15.2-debug
232     /nix/store/bmzabsyqg5zgw908aga2xlyl5s36dabm-python2.7-pyodbc-4.0.25-debug
324     /nix/store/1lss2a7h9swfc9z22jq01qvfimsl6m0m-epdfview-0.1.8-debug
380     /nix/store/cqgn8g526hp49pbywkv9sr1g619sagz4-kmediaplayer-5.54.0-debug
588     /nix/store/5s7862r6fljwh2fxs5683g938jxb9aqc-python3.7-python-imread-0.6-debug
596     /nix/store/s3xpkhqqyb4l64fag7gprvpxg3bf4agn-python2.7-pygobject-3.30.4-debug
608     /nix/store/9ys9shyjpbnjyldk660g10ln1k9ha17n-can-utils-20170830-debug
732     /nix/store/rjzmn88g157qy15zndng3xdj21wqnh00-libtcod-1.5.1-debug
1164    /nix/store/bgk1bya00p95in9w1qrpy9l5cclw1sgl-swh-lv2-1.0.16-debug
1456    /nix/store/8yl8bmqcb93ixnk2qzm00jbamlsn4c7j-klibc-2.0.4-debug
1536    /nix/store/csp7ijbmk7j01297nqnw8l0n5xcirnri-python2.7-mpi4py-3.0.0-debug
2632    /nix/store/p8mcfhkgc2176m7kfdsib9wfaq6fj0qi-cargo-update-1.5.2-debug
2904    /nix/store/gzii09fj42ik09czvnqxbph6cgx6iywx-vdr-fritzbox-1.5.3-debug
3008    /nix/store/m6na0qsj7dbs1xkdby1h8rfnq6q6bnjq-neovim-unwrapped-0.3.4-debug
7964    /nix/store/560zq6kzy5za1bslpzrl1q41dmw5csfm-eventviews-18.12.1-debug
15924   /nix/store/6lq9slb05ndbx6g5b9g2ads25hr8sink-xqilla-2.3.4-debug
48004   /nix/store/4m07q2v2l34a62njwxg3y0ap4r8czfqd-openbabel-2.4.1-debug

EDIT: well, perhaps it would've been better to sample from *all* the binary cache paths instead of just those from a single channel release, as different packages will have different rebuild "rates" that might well be significantly correlated to the ratio of their .debug size. The problem is that I do not have that data, so I couldn't do such sampling.

nh2 commented 5 years ago

I suspect this flag isn't enough for haskellPackages, as their sizes seem suspiciously tiny.

@vcunat Yes, you need a DWARF-enabled Haskell compiler. I see you already found that info here, but you may not have seen that I had already gathered some basic statistics: https://github.com/NixOS/nixpkgs/pull/52255#issuecomment-447646279. In that case, the DWARF overhead was 3.5x for this large Haskell executable, and 3x when xz-compressed.

vcunat commented 5 years ago

Apparently for just haskellPackages the total extra cost will be larger than for the rest of NixPkgs. Very quick estimate: 30–40 GB, based on the haskellPackages subset of the list above (26 pkgs) and code from the other PR (which does not split an output).

Moredread commented 5 years ago

How do these numbers compare to builds without debug symbols?

vcunat commented 5 years ago

I'm not sure ATM. I only tried to estimate the .debug sizes so far, not the size of the rest.

edolstra commented 5 years ago

We should set up a jobset with debug symbols enabled globally so we can compare the total size with nixpkgs master.

vandenoever commented 4 years ago

Debug symbols for large packages like Qt, Firefox and LibreOffice would be very welcome. The amount of work a user needs to do to write useful bug reports for upstream is unrealistic.

I do not know how to enable debug symbols for libraries. I figured out that for leaf packages I can add an overlay like so:

self: super:
{
  kcalc = super.kcalc.overrideAttrs (oldAttrs: rec {
    separateDebugInfo = true;
  });
}

in ~/.config/nixpkgs/overlays/. But that just lets me debug the top layer of the stack.

To send useful crash reports to projects or do debugging, debug info for the entire stack is needed.

How can I enable debug info for all Qt and KDE libraries?

grahamc commented 4 years ago

I would like to revisit enabling debugging symbols for all packages. From what I've read and heard, doing so will have some negative effects:

binary cache storage will go up significantly
build and upload time will be lengthened, increasing load on the Hydra coordinating machine

I don't have a solution for the second point, but for the first: would it be valid to periodically trawl the debuginfo directory in the s3 bucket for files older than 1 month, and delete the NARs associated with it?

Would this eliminate the first concern, by keeping only a rolling set of debuginfo files?

edolstra commented 4 years ago

We could revive the binary cache garbage collection script. That would reduce storage costs a lot, not just wrt debug symbols. Otherwise deleting NARs referenced by the buildid links would be dangerous (you don't want to delete something reachable in some other way, and debug symbols don't have to be in a separate output).

Ericson2314 commented 4 years ago

I think we're making a mistake conflating nixpkgs and cache.nixos.org policy here. Even if we never stored debug outputs, but still built them, it would be useful to CI that the building still works. I'm OKish doing a mass rebuild for myself to get those outputs, but only if I'm sure that mass rebuild won't be in vain because it isn't tesed and bitrots.

poelzi commented 3 years ago

Debugging on NixOS is one of the most frustrating experiences ever and drive me to the point where I want to ditch NixOS again. Debug symbols for everything need to be install-able and working for all packages and especially libs without days of trial and error. Status quo is just broken and useless, better debug your stuff on a debian/arch/whatever VM then try to get this stuff on nix built...

Gaelan commented 3 years ago

Would it be possible, as a compromise/interim solution, to have Hydra build packages with debug information enabled (as an extra output), but then ditch the extra output as soon as it's built? That would eliminate the disk space requirements on Hydra, but since things would be in the "with debug info" store paths, it would be possible to obtain debug info for a Hydra-build package by "simply" rebuilding the package locally. We could even have third parties/community members distribute debug info for some packages if there is interest.

stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

oxalica commented 2 years ago

I had a hard time when debugging firefox issues recently due to the lack of debug symbols.

I also ran into the issue of separateDebugInfo not recognizing artifacts from LLVM toolchain when trying to enable debug symbols for firefox. See #146275

fzakaria commented 2 years ago

Want to chime in here.

I went through the process of building glibc with debug symbols to triage a linker error. The documentation in https://nixos.wiki/wiki/Debug_Symbols was amazing; as an aside it should definitely work it's way into the official documentation.

I definitely agree that conflating building by default all debug symbols and whether or not to store them are two separate things. Having potential automatic debug symbols for NixOS / Nix would be such a game changer, and would also highlight the power of the fact that it's a source based build system.

Perhaps even the $$ discussion can be revisited. Hopefully after the years since this ticket has been open, the monetary outlook and funding of NixOS is different and this can be reviewed.

Doing this for only x68-64 to start is also extremely sensible.

Two improvements I could see as well:

enableDebugging that also perhaps turns off "separateDebugInfo". Sometimes it's easier just to have it in the binary
enableDebugging that also applies itself transitively to all build dependencies of the package.

<3

lheckemann commented 2 years ago

I'm inclined to agree with the approach of not keeping and serving debug symbols other than for packages like glibc. The missing piece for this, as far as I know, is support in hydra for only keeping a subset of outputs. Perhaps this would be best achieved by an extra meta field which defines outputs to discard or outputs to keep?

layus commented 2 years ago

Discarding debug outputs seems the best way to go for now. Adding them to the cache for a fixed amount of time may come second, as a nice-to-have feature. So :+1: for me.

We should however keep in mind that it is not perfect. There is a risk of franken(debug)builds, namely the possibility that you associate a locally re-generated *-debug output with a cached output, and get mismatches because of build impurities. This would not happen if all the builds were perfectly reproducible, but that is not the case. See https://gist.github.com/layus/2a91694be330d5f2591dab9a52c47548 for a mwe.

Nix managed to avoid showing that issue by never garbage collecting its cache (!), and always uploading the full closure and outputs set of any cached path.

NixOS / nixpkgs

Enable debug symbols for all packages #18530

Notes:

Methodology, more numbers