bazel: shrink output size / closure

[Profpatsch]

$ du -hs /nix/store/8r49ijzjpc096ixzwk62vykr1h9rranq-bazel-0.15.1
102M    /nix/store/8r49ijzjpc096ixzwk62vykr1h9rranq-bazel-0.15.1
$ du -sch $(nix-store -qR /nix/store/8r49ijzjpc096ixzwk62vykr1h9rranq-bazel-0.15.1)
…
576M    total

We should trim that down, there is a design document on how to do that: https://docs.google.com/document/d/1Igmv-2GfXkoVFWTXvBYPeniQom8nLAwzqzridDlBIS4/edit#

cc maintainer @mboes

[mboes]

Is this something for us to do or can we just wait for upstream to deal with this?

[Profpatsch]

Since bazel ships with a full jdk, building the minimal JDK that is mentioned in the document could reduce the size by about 50 MB if I read that correctly.

[uri-canva]

We already have jre_headless, if we had jdk_headless we could use that for Bazel.

[Profpatsch]

For reference, we’re up to 735M for 0.17.1

[rdnetto4]

Since bazel ships with a full jdk

This isn't true as of 0.19 - see the Java section of the changelog.

I believe that means that this issue can be closed once we update to it.

[uri-canva]

I think even with 0.18 Bazel uses the embedded JDK for running, so the closure doesn't need the jdk, we only need the jdk to build Bazel, not to run it.

Is it possible to declare a build time dependency in a derivation and exclude it from being a runtime dependency in the closure?

[Profpatsch]

If the --javabase flag is unset, Bazel locates a JDK using the JAVA_HOME environment variable and searching the PATH. If no JDK is found --javabase will be empty, and builds targeting Java will not be supported. Previously Bazel would fall back to using the embedded JDK as a --javabase, but this is no longer default behaviour. A JDK should be explicitly installed instead to enable Java development.

From that description it is not clear to me that the embedded JDK isn’t used at runtime for the bazel tool. It only describes that no fallback for compilation of Java code with bazel. Does somebody here want to figure out whether that is true or false?

[rdnetto4]

From that description it is not clear to me that the embedded JDK isn’t used at runtime for the bazel tool. It only describes that no fallback for compilation of Java code with bazel.

AIUI, Bazel still has an embedded JDK (for running with), but the motivation for the change is so that they can use a cutdown/minimal JDK rather than a full-size one.

[uri-canva]

Note that at the moment Bazel still builds without the embedded jdk by default, so the first step for not having the jdk in the closure would be to change the Bazel derivation to use the target of Bazel with the embedded jdk: https://github.com/bazelbuild/bazel/blob/3dd1309d04621df185cf630b65d4da10715ee768/compile.sh#L95

[Profpatsch]

du -sch $(nix-store -qR /nix/store/8r49ijzjpc096ixzwk62vykr1h9rranq-bazel-0.15.1)
1.5G    total

13M /nix/store/msvdi3zrfdxah6arc1gahi9m2rv9qdci-glib-2.62.1
22M /nix/store/rxn6ai6xbsp2qzims27s8mwx700ywkg8-ffmpeg-3.4.6
25M /nix/store/xclhhlajs3ylzp40hlyrchk6z4vjvc38-spidermonkey-60.9.0
26M /nix/store/zq8qbl2hp3vvcbc5hf9gisvj5y0kg9qj-bazel-1.1.0
28M /nix/store/pa29pw2k0kzwr8d60xndmaih2wx5i0xl-gtk+-2.24.32
29M /nix/store/qn76sklvyalzw9ilnxz6sh0020gl2qn6-glibc-2.27
33M /nix/store/713z2dlgbi0nspi5hyl6hbf0m2kr7li3-icu4c-64.2
33M /nix/store/cg0k49h66nkdqx6ccwnqr0i4q0fnfznc-binutils-2.31.1
36M /nix/store/a6jdq1g4ly7s2rwxyiywmkccdq92i4v4-gtk+3-3.24.12
40M /nix/store/ndh6jr1hv1iwqmdi6gzm2bj0m1fq8bv5-bazel-deps
46M /nix/store/jwjgz37zw6spgjhg92m9cbkv98vjfywk-java_tools_javac11_linux-v6.1.zip
48M /nix/store/bzf8x5kpv3x4qi09xq2grnvccakvy7gz-python-2.7.17
54M /nix/store/zdh16dcvjw99ybam59zd2ijb6bx138j0-python3-3.7.5
65M /nix/store/mhxqp4fabz9yq6wh20pz9ngc9mlfk35c-remote_java_tools_linux
137M    /nix/store/yvzq185jfr4vj74a9h4ap4dj4nnqbx7l-gcc-8.3.0
674M    /nix/store/g6rf6bf1wv9jkbnz4lv33givkgaz06j0-openjdk-11.0.4-ga

[uri-canva]

On macOS:

> du -shc (nix-store -qR /nix/store/mp9jsqihzpb80qajkrqi3vwp6xkhj1j2-bazel-2.1.0)
1.7G    total

 11M    /nix/store/c9wjlsjx1xkbljsvk90npmj76phnvch9-binutils-2.31.1
 20M    /nix/store/42svdgwl6zqyzgsqbfmw9rh7bssyms0a-Libsystem-osx-10.12.6
 25M    /nix/store/mp9jsqihzpb80qajkrqi3vwp6xkhj1j2-bazel-2.1.0
 44M    /nix/store/4zlj6xw1bi4048axk5hxkbc3xph0v5hi-java_tools_javac11_darwin-v7.0.zip
 53M    /nix/store/8lkm3zd71xavhrpskd2iwmw1a8hkhnja-python-2.7.17
 60M    /nix/store/4m82hrq0xgi67j1wj53sr0bl2c0lf5av-llvm-7.1.0-lib
 63M    /nix/store/1s66apwyh9ia021xigf19bvjw7krhqxf-python3-3.7.6
 65M    /nix/store/ry2q6pf7acs19n4vlmh7203qs5237cs8-remote_java_tools_darwin
 71M    /nix/store/h6kwdhhsc15dg2xc9kin834wiv2zc6vb-clang-6.0.1-lib
 85M    /nix/store/kal84pakd6g3wl5l7ws125m1al7sc5pv-bazel-deps
130M    /nix/store/6559s991g16wqhj6kyfa2am0qyy0p09s-llvm-7.1.0
175M    /nix/store/0fjmxmjsf2d8agqqwwa2ihrr7vrq0czr-gcc-9.2.0
309M    /nix/store/6hrbn46g3wwjsdgs8nhvn0cyd9d3rzyv-zulu11.2.3-jdk11.0.1
499M    /nix/store/cr2m6byfa10hlih556ksli7mj3y15i6h-clang-6.0.1

[stale[bot]]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

[uri-canva]

Still revelant:

> du -shc (nix-store -qR /nix/store/5pml7q400msg64idgl6708gjmni6w7zg-bazel-3.3.1) | sort -h
  0B    /nix/store/ip78z0sjkhafnqq6zckm3zx4n3lcf3q3-cctools-binutils-darwin-927.0.2
4.0K    /nix/store/4ygqr4w06zwcd2kcxa6w3441jijv0pvx-strip.sh
4.0K    /nix/store/cickvswrvann041nqxb0rxilc46svw1n-prune-libtool-files.sh
4.0K    /nix/store/fyaryjvghbkpfnsyw97hb3lyb37s1pd6-move-lib64.sh
4.0K    /nix/store/ggalj2rbpkrfhsyc25ji0ai8kslkg8ka-hook
4.0K    /nix/store/kd4xwxjpjxi71jkm6ka0np72if9rm3y0-move-sbin.sh
4.0K    /nix/store/kxw6q8v6isaqjm702d71n2421cxamq68-make-symlinks-relative.sh
4.0K    /nix/store/li3nal12j4l4qypwvxkyw8nm200ipj4c-bazel-rc
4.0K    /nix/store/mjjy30kxz775bhhi6j9phw81qh6dsbrf-move-docs.sh
4.0K    /nix/store/ngg1cv31c8c7bcm2n8ww4g06nq7s4zhm-set-source-date-epoch-to-latest.sh
4.0K    /nix/store/rvg5a5nwa7cihpmbzlwzh931w3g4q108-compress-man-pages.sh
8.0K    /nix/store/1i5y55x4b4m9qkx5dqbmr1r6bvrqbanw-multiple-outputs.sh
8.0K    /nix/store/60wapxrdcmf2nrs3q4admry485kzgarg-auto-patchelf-hook
8.0K    /nix/store/6k829xz9v508ncxfcf2p1b8g1wq54946-patch-shebangs.sh
 12K    /nix/store/hj445avh81v7ymydvi3xin9nfc8q7v89-expand-response-params
 12K    /nix/store/mi7i65p00hgvfzxyx6y0kzzzjkjhyygq-expand-response-params
 12K    /nix/store/wa3mca8ixs2llzdcl0xhjxcisn8aak79-bash
 40K    /nix/store/5wg1d0rd6b5xl91si1i5brlylb348bkr-stdenv-darwin
 40K    /nix/store/yn0n6sirg3d3x2by9ynvwjrhpf2npiip-which-2.21
 44K    /nix/store/1d3laa1r1vv04wydp5zgkc80ay4ysrqi-libffi-3.3
 56K    /nix/store/igxxil6llh8sz3wg4mm19hja98sfd7ms-cctools-binutils-darwin-wrapper-927.0.2
 68K    /nix/store/wdrrx3gkbljwq1v1wywdpdaihbc7mblj-bzip2-1.0.6.0.1-bin
 80K    /nix/store/bafwypxc11wp51phssmh9h6vqmwzmnxf-bzip2-1.0.6.0.1
 88K    /nix/store/z5fnc4yx39kasncdz08dzv8sdpd58gzx-clang-wrapper-6.0.1
100K    /nix/store/ilrm722frr4fb3096wjc6wbspdf7pgyr-zlib-1.2.11
120K    /nix/store/li3hxwxsljalx0y7yvvw21kkz6q97yxx-zlib-1.2.11-dev
132K    /nix/store/ydairwanls4gjkalyab3b3h5sj53jv47-ed-1.16
144K    /nix/store/cfbxf47rvwff3mj94b35m8j6mdl9nv57-xz-5.2.5-bin
152K    /nix/store/6xqpvdkp16vq17rgp300sa83hwwkgmmw-xz-5.2.5
164K    /nix/store/6zqj4cq0dlnw2212hfqb5h79c2jqyca5-nghttp2-1.41.0-lib
180K    /nix/store/rv4c0bc8amcrxdahnbqafr1wd30f1sim-gzip-1.10
200K    /nix/store/9a93ycw78s472h3cw15ijhz73vx28cad-libmpc-1.2.0
208K    /nix/store/cjflr1cvpkq74ln7j2qndri3a5513797-expat-2.2.8
216K    /nix/store/rqjf0pgm9n9xn0844xprxnqp2slg35mb-gnused-4.8
220K    /nix/store/zrgd2dyrny2il2zwv593karmrnzvcslw-gnumake-4.3
228K    /nix/store/5xim5vyzv20wb6z06nn7l6fm01ir43l8-patch-2.7.6
236K    /nix/store/myqr5s2aiz55nf5pg6labf3y9zz1ihzi-libssh2-1.9.0
280K    /nix/store/qczfddpm9038gk00in28b19x58zh37jq-readline-6.3p08
280K    /nix/store/sr8lgii6aslxxjv3h33za80ni305k6np-gnugrep-3.4
304K    /nix/store/8yhdhkig2w1yzlg859lq33k0k7fl4vn4-compiler-rt-6.0.1-dev
376K    /nix/store/pcr57a7awj7y3a33giwrmvc1s6r67cvj-diffutils-3.7
408K    /nix/store/5sljvp3yjl8d3hhdpmvh48iilfvc91yz-gdbm-1.18.1
432K    /nix/store/hfkh7680z5wm0kpsqglkw5diysdsrrpk-ncurses-6.2-dev
456K    /nix/store/sfppw36j1y5yr3pjp48nzca004rxrqdg-mpfr-4.1.0
500K    /nix/store/k0ws65jj21wyq297pv8ammljh7aywyvw-unzip-6.0
524K    /nix/store/5yvd7mxwdzd287ar3gh6scl9z8w9dnww-libc++abi-6.0.1
532K    /nix/store/l1zifrb5i2w4jhag760xckmcm48b50fx-libc++abi-7.1.0
532K    /nix/store/yqbfysnlz9irkn2r1xrh2nig41sw90nw-curl-7.72.0
544K    /nix/store/40rbdr2chhsx3fqsr6v9li1lkxaix3ww-pcre-8.44
568K    /nix/store/vx92f3649cwh35ljanpz7352qq035i5g-zip-3.0
668K    /nix/store/9y1lgfkzi22jnw5x3blmi1nycrlj7hmf-ncurses-6.2-man
668K    /nix/store/xg4310nmfd8a1gkwgn1g178v0jp7913k-findutils-4.7.0
692K    /nix/store/p20ys2ryai7h3hm2bihbk137is598k9z-gmp-6.2.0
968K    /nix/store/im427hd05ck38dyi6rwcyslw8z3n82ip-gawk-5.1.0
1.1M    /nix/store/dkyh7ygmqglmvscn9ikx7b64l1hcnxjq-configd-osx-10.8.5
1.1M    /nix/store/m0a4283w5mzspfcx353633262kaqn5d6-bash-4.4-p23
1.1M    /nix/store/yx90ay96pjn5vckwivzmbiqyxlyymgml-libiconv-osx-10.12.6
1.2M    /nix/store/yal84bhmgpqw4fqsyxv6yh8nb37cf019-coreutils-8.31
1.3M    /nix/store/s3hxlbhkr5b3kr63x82ssl23yygiv3wn-Security-osx-10.9.5
1.4M    /nix/store/z08x4fwkv7r7gjvzb8svvmrhskrwzzb0-libxml2-2.9.10
1.6M    /nix/store/b6fhlbpyyn05wgl82876vpnbv3mhi8m0-sqlite-3.33.0
1.7M    /nix/store/0p56hc1phcdw0i33zfr9x8hy3pqsx87f-libkrb5-1.18
2.3M    /nix/store/7jmm1dbx81l95vrzqqsnp5ls23l75046-adv_cmds-osx-10.5.8-locale
2.7M    /nix/store/8dh1f9qfd554dzpbm748lmxy5vihv9rk-gnutar-1.32
2.7M    /nix/store/mblj5njr6ja3y3815clvq4ny3s61nngs-gnutar-1.32
3.1M    /nix/store/3g5405skrdb1ajsvaix4al1v2jsy49zf-openssl-1.1.1g
3.1M    /nix/store/whmwp70qvfhl4yf2vgrcxpld20h3nyxq-openssl-1.1.1g
3.5M    /nix/store/rc7bjj6rq9hi3p3ylpxvcx5yv3qn0hfk-swift-corefoundation
4.2M    /nix/store/vpgdckiiiyw43pvlqhqzs36i70shbhlm-gcc-9.3.0-lib
6.6M    /nix/store/wzina4gcrg38pq5k5j0pv4p254wwc4nk-file-5.39
6.9M    /nix/store/7pcrpzwhqpm9w2lfzrxx40g7p7j90yx4-libc++-6.0.1
6.9M    /nix/store/dp5wz79759zxa94k29bdw3arvpkbbhjv-ICU-osx-10.10.5
7.0M    /nix/store/fxynv58g4w6fha7yamkk13cnkcdymc4b-libc++-7.1.0
7.5M    /nix/store/wcx8ps9hpaprs82wwfi518rvqrxrv04y-ncurses-6.2
7.6M    /nix/store/d0mp55cyqs5a0rmg3wp8zn3f1g6jn43x-compiler-rt-6.0.1
8.5M    /nix/store/vrxrz8fqi2cmv0z7kxfza2nycbbbkh1k-gettext-0.21
8.6M    /nix/store/k76hy0421bdillj1q97hal0nc1q9akc4-gettext-0.21
8.9M    /nix/store/adz1mhlaj4jg7zf7qlpqq34m7yk18s4n-cctools-port
 11M    /nix/store/yznnkkmpvw9860khghck3839dbs26j1c-binutils-2.31.1
 20M    /nix/store/hw4x6gakgd43f13zys92ng0hj0acisls-Libsystem-osx-10.12.6
 27M    /nix/store/5pml7q400msg64idgl6708gjmni6w7zg-bazel-3.3.1
 48M    /nix/store/0mc2149zvgcqaqkfakxm5rl0pz45mc63-java_tools_javac11_darwin-v8.0.zip
 53M    /nix/store/xifgh5qz2pvafhxcfvldgpa44yk7mppw-python-2.7.18
 61M    /nix/store/p0pgpgr0jgl4sx0dnai9cr2bjqfzq615-llvm-7.1.0-lib
 66M    /nix/store/np0b7p04kbhl244yv65d2sc37kk7wcqf-python3-3.8.5
 68M    /nix/store/33zz58h6fv1pmvmnrs2f5s6br052njdq-remote_java_tools_darwin
 81M    /nix/store/989w0fxszx13jka49hbjqbsdj253vbya-clang-6.0.1-lib
 89M    /nix/store/klr182h3snlfzhrxvam4099n98308cng-bazel-deps
130M    /nix/store/df471kxb4l71y4fvjcj8q5naqvnzi519-llvm-7.1.0
173M    /nix/store/gp8fsq6c1bsjxxiag15zci0x84g7pnj1-gcc-9.3.0
308M    /nix/store/h6kh5nhcqp4snxv3xzkx7z632h2n0f5q-zulu11.2.3-jdk11.0.1
507M    /nix/store/3z6c3qmsqjmy8vdgfb3wi3j2mkgvj3zn-clang-6.0.1
1.7G    total

[stale[bot]]

I marked this as stale due to inactivity. → More info

[uri-canva]

The closure just keep getting bigger and bigger as we add more dependencies to satisfy bazel's toolchain autodiscovery. This is especially wasteful for people using https://github.com/tweag/rules_nixpkgs or other mechanisms to specify toolchains. It would be nice to have a different version of bazel that didn't have the toolchains, like a bazelMinimal.

[Profpatsch]

Agreed, if possible with a simple mechanism of adding the subset of toolchains you need by overriding or using a bazel.withToolchains function.