search

easybuilders / easybuild-easyconfigs

A collection of easyconfig files that describe which software to build using which build options with EasyBuild.
https://easybuild.io
GNU General Public License v2.0
379 stars 701 forks source link

Make Bazel builds (e.g. TensorFlow) disconnected #11174

Open Flamefire opened 4 years ago

Flamefire commented 4 years ago

https://docs.bazel.build/versions/master/external.html#offline-builds shows that Bazel can be made to not download anything by passing --nofetch

Using --distdir one can download all files manually before just like the git submodules for PyTorch

moreover, the using the option --nofetch, fetching of further repositories can be disabled during the build.

For true offline builds, where the providing of the needed files is to be done by an entity different from bazel, bazel supports the option --distdir. Whenever a repository rule asks bazel to fetch a file via ctx.download or ctx.download_and_extract and provides a hash sum of the file needed, bazel will first look into the directories specified by that option for a file matching the basename of the first URL provided, and use that local copy if the hash matches.

There is also Gentoo which packages TF and its dependencies. E.g. https://gitweb.gentoo.org/repo/gentoo.git/tree/sci-libs/tensorflow/tensorflow-2.2.0.ebuild https://gitweb.gentoo.org/repo/gentoo.git/tree/net-libs/grpc/grpc-1.28.1-r1.ebuild

Looks like they use that too:

# distfiles that bazel uses for the workspace, will be copied to basel-distdir

casparvl commented 3 years ago

I struggled doing offline bazel builds recently. Never got it to work, since there doesn't seem to be any (convenient) way to download all the tarballs that you'd need to provide in --distdir. I put in a feature request, at Bazel https://github.com/bazelbuild/bazel/issues/13493 Let's hope something happens there... :)