Darwin stdenv bloat

[edolstra]

The Darwin stdenv has become crazy huge, about 3/4 of a gigabyte:

$ du -sck $(nix-store -qR $(nix-build -A stdenv)) | sort -nr
743184  total
283664  /nix/store/479h7d7zvg6a6yq3rz5hycakz9qarmpb-llvm-3.6.2
213020  /nix/store/cn5mg38x2ijvxl93hbn69g2jc6fn71hc-CLTools_Executables_OSX109
176800  /nix/store/3ipm9yr9jz4mcgc9n2fa85rwrjjk2bi0-clang-3.6.2
15120   /nix/store/1p55mnh9jzhqpyxrw4aabp5jrg2ggcl9-ncurses-5.9
10756   /nix/store/kpwz5skpq9xmr0pkrllihnh823fmlfvk-coreutils-8.24
10156   /nix/store/dp60hwixrs88aqvgv86va70b9qlfhlqc-gettext-0.19.5.1
7944    /nix/store/vl87cnscn1lmysfqc9h84jr0pfh1715a-cctools-port-862
5096    /nix/store/dg8absgv0q0ri929wnzmbn64n5ni0ami-bash-4.3-p39
4780    /nix/store/whdp09gkfw7jvjp1l6i746cncf83p901-libc++-3.6.2
2960    /nix/store/krm67ws2d6cips6i5xkablyjsbzwry3g-gnutar-1.28
2724    /nix/store/0gfdlafrqgggv1xqhpvpqijvj6rij50g-gawk-4.1.3
2308    /nix/store/x0vqly4g6km9gfyz7yl4gq7qq3k8s7ri-gmp-5.1.3
1280    /nix/store/hzhnh50vzfhqa7rsqgwbh9sl2f01br97-findutils-4.4.2
1116    /nix/store/waj9y9ijh5wr9svy8wlb2v670xvd0d3z-libiconv-41
...

The clang runtime dependency on LLVM seems preventable. Or maybe the size of llvm can be reduced (e.g. by moving static libraries into a separate output).

CLTools_Executables_OSX109 contains such stuff as Subversion, Git, and another copy of Clang, so it should be possible to reduce it significantly.

Graph of closure size history: http://hydra.nixos.org/job/nixpkgs/trunk/stdenv.x86_64-darwin#tabs-charts

[copumpkin]

Damn! cc @pikajude @gridaphobe. Is this the pure stdenv or the other one?

[copumpkin]

As of 506b491edd4937df63a2594a5d14b86d9e870710:

$ du -sck $(nix-store -qR $(nix-build -A stdenv)) | sort -nr
718680  total
347684  /nix/store/9k0dfmgp3wlnzil6m4dgbg1n948drb3b-llvm-3.7.0
236016  /nix/store/8gwqwvn2k1nfvnw3xz4n5ymg5gdmgpg5-clang-3.7.0
37520   /nix/store/b4rrhhs533ir7z5f82m8haa8qdr8v36f-icu4c-56.1
15328   /nix/store/p427bw6awm4rv8x3dph4za6gynp3062d-ncurses-5.9
10776   /nix/store/9fzrf9wripj4fybwn09c3xzvkwh25qc1-gettext-0.19.6
10596   /nix/store/jiwdg4gvlvkigk3dkzpxyxhnr2i2y7y5-coreutils-8.24
9920    /nix/store/vcv4hb3isw7rp7grc725skjjpblpsgd8-binutils-2.23.1
8352    /nix/store/kn1i9a6si3kskj25vsmv0ihh9rfgrdi4-Libsystem-1197.1.1
8008    /nix/store/7fi5j0hhb60arwrjk3cvl3ai6g7kydza-cctools-port-862
5632    /nix/store/ambhwjp4yr6bfg6k52iiv8hkmrd5fycm-adv_cmds-119-locale
5088    /nix/store/nnhl5b4mzgsmincz05jns5ngx2mjsdai-bash-4.3-p42
4828    /nix/store/b5h38zwyq2f9h2xcik30inyklb5sj5pl-libc++-3.7.0
2964    /nix/store/v5jqgmb409cwq6621bm9pp29nza9av00-gnutar-1.28
2724    /nix/store/bzf0w3v8b2x3dbbp0rmhjzd0anvg8l55-gawk-4.1.3
2528    /nix/store/50zlccqm128p16gqj9ax6zyds1pnsv52-CF-855.17
2316    /nix/store/x967xnkzw09zfwf05b5q1cxg0mx8f65w-gmp-5.1.3
1276    /nix/store/c5dimkrkr90qlqw9bzms34w6cr91cj2f-findutils-4.4.2
1116    /nix/store/fywq42mjmivkh4g1i1lq5kc30dyl8kjn-libiconv-41
848 /nix/store/74svaq49chabsbvvwvzajqr7ijdh6rd3-libc++abi-3.7.0
824 /nix/store/5wr970jr8p3wqi3ydzy3yvr07a11qjsn-gnumake-4.1
784 /nix/store/kwmil65qvqbbayyba332wsc76wbb0r7c-pcre-8.37
676 /nix/store/2dsczipb5amaknzbf05w7j0h01fs7zyr-xz-5.2.2
592 /nix/store/144cib3svkqpqavhc36l9fl1jjbbg78d-diffutils-3.3
400 /nix/store/bj98wr6m1cqnk20m01m282zaaxf1nm3q-gnugrep-2.22
328 /nix/store/gmzglzhdgbjx80nizcxxyy5r8hp072d3-zlib-1.2.8
288 /nix/store/q8zs47h8f4jxnqvs2814cvshsiz4z1fn-gzip-1.6
280 /nix/store/yicwy6dl04qpzrg71i8dyg3swk11jv8x-gnused-4.2.2
280 /nix/store/j5mz4ka98ifb3qh43rppnfh0s00cx8fl-bzip2-1.0.6
212 /nix/store/2zd9ygyh9wiwp31kj4m5f5rziskshf3h-patch-2.7.5
132 /nix/store/65mz1g0w9iydcvdz9lwfhdpmz3kb5778-ed-1.12
120 /nix/store/kn310xd85lv0dagk811a9mrx066zb36l-libffi-3.2.1
100 /nix/store/cxq16hncmgf4pmdc21hcdil4fxwdxa4c-cctools-binutils-darwin
92  /nix/store/rgpxhfsd8xivvqqhxgqi3kqwm1sciwpm-clang-wrapper-3.7.0
28  /nix/store/n2c5jzbpzlylw86whyfdsp430qp5c6ql-stdenv-darwin
4   /nix/store/zr8kv6m6565y3l7dc8rcc2cvwfmlv92v-compress-man-pages.sh
4   /nix/store/z82dl6ialp166drqihzkz67nkl6w3l16-move-sbin.sh
4   /nix/store/a92kz10cwkpa91k5239inl3fd61zp5dh-move-lib64.sh
4   /nix/store/7ps94j28m8jgrb4za09gckl37my915v1-patch-shebangs.sh
4   /nix/store/5hzb6zwq8xls6ny9pzg8h8rmp0p903y1-strip.sh
4   /nix/store/17h0mw5sipbvg70hdsn8i5mai4619l8c-move-docs.sh

So the pure stdenv gets rid of the CLTools, but the llvm issue is still there. I'll see what I can do about it.

[matthewbauer]

Would it help to make llvm and clang have multiple outputs?

[Profpatsch]

(triage) is this still the case?

[matthewbauer]

Yes it's still 500M+.

[vcunat]

Around 700 MiB on 18.09 ATM.

[copumpkin]

I think a big part of it is that clang (and probably LLVM), as best as I understand it, statically links (at least doesn't dynamically link to libclang) all its binaries:

$ du -sh /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/* | sort -h
0   /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/clang
0   /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/clang++
0   /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/clang-cl
0   /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/clang-cpp
0   /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/cpp
56K /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/scan-build
1.9M    /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/clang-apply-replacements
2.0M    /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/clang-format
2.8M    /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/clang-offload-bundler
18M /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/clang-include-fixer
18M /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/clang-rename
18M /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/clang-reorder-fields
18M /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/clangd
18M /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/find-all-symbols
18M /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/modularize
19M /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/clang-change-namespace
20M /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/clang-query
21M /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/clang-import-test
25M /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/clang-tidy
56M /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/clang-check
71M /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/clang-5.0

$ otool -L /nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/clang-5.0
/nix/store/wbr183a4p1mip7k9wrdv089qw07jnh4n-clang-5.0.2/bin/clang-5.0:
    /nix/store/da06jygxfr71v2x308c130awmawsp1zh-ncurses-6.1/lib/libncursesw.6.dylib (compatibility version 6.0.0, current version 6.0.0)
    /nix/store/2vv68rv8p3kxzvy4kv3r5snl96wzi0r0-zlib-1.2.11/lib/libz.dylib (compatibility version 1.0.0, current version 1.2.11)
    /nix/store/iricrrlps2jilc13a4ry5b6q1iwsl6j7-Libsystem-osx-10.11.6/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
    /nix/store/4850ibf30qpr5nx81hc94r4npyqd76sz-libc++-5.0.2/lib/libc++.1.0.dylib (compatibility version 1.0.0, current version 1.0.0)

Which leads for example to a 320MB clang bin directory, but we still depend on the 60MB clang-lib derivation for other reasons.

Also binutils seems do something similar: even the smallest executable in it is 6.5MB which suggests that they're probably statically linking in libopcodes and/or libbfd.

It seems like if we could switch clang, llvm, and binutils/cctools to use dynamic linking of executables, we'd lose over 50% of the closure size. I won't have time to try it out for a while but if any of you are interested in trying, it's probably just a flag to their respective build systems.

[vcunat]

For binutils see #46056.

[copumpkin]

I spoke to the LLVM folks a bit and it turns out it's trickier than expected for LLVM and clang. They don't really support dynamically linking against libclang from the clang binary, for example, and say that even dynamic linkage against libLLVM can slow things down nontrivially. They did however say that our binaries are unusually big (beanz said his toolchain was about half the size of ours, even with static linakge). I'm wondering if this is somewhere the lack of LTO is biting us. Not sure what else would cause such a discrepancy.

[vcunat]

For the use in compiler it seems OK to have all static, even including LLVM. The compiler shouldn't remain in closures, and this time-space tradeoff can actually help on Hydra (in Darwin case).

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.