search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.34k stars 14.3k forks source link

buildBazelPackage: make dependency injection possible #289015

Open SomeoneSerge opened 9 months ago

SomeoneSerge commented 9 months ago

Issue description

From what I see, the way we package jaxlib and tensorflow is unsustainable. One of the symptoms is the deeply nested conditional expressions for the hashes of the vendored dependencies (fetchAttrs). Besides, fetchAttrs is potentially a bootstrap, package set consistency, and security issue.

Consider if it's feasible to automatically generate something like "bazel repositories" that ingest pkg-config/cmake exports from buildInputs, and if it's possible to substitute bazel's "source builds" of the vendored dependencies with the ingested targets in a generic manner

Possibly a relevant discussion upstream, limited to bzlmod: https://github.com/bazelbuild/bazel/discussions/20581

CC @NixOS/bazel @zeuner @layus @aaronmondal @NixOS/cuda-maintainers

samuela commented 9 months ago

Possibly related: bazel includes a --override_repository flag that allows one to override the vendoring of third party code (eg pulled in via http_archive, local_repository, etc). This could be a nice alternative to some otherwise pretty hairy patching. A notable caveat being that this requires the source, not build artifacts, meaning that although we could abstract the XLA source out of both the JAX and TF builds, we would repeat the XLA build twice as a constituent of both the JAX and TF builds.

The other issue being that I have yet to get it to work successfully in jaxlib :P

uri-canva commented 6 months ago

I think we can do this in a sensible way for packages that use bzlmod.