bazelbuild / bazel

a fast, scalable, multi-language and extensible build system
https://bazel.build
Apache License 2.0
23.31k stars 4.09k forks source link

ENH: equivalent of `local_cc_dependency` override in bzlmod #20947

Open h-vetinari opened 10 months ago

h-vetinari commented 10 months ago

Description of the feature request:

Hi!

I help maintain the conda-forge distribution, where we clash pretty fundamentally with bazel's philosophy of hermetic builds -- we absolutely cannot rebuild all dependencies in a single build, it MUST be possible to leverage libraries we have already built for the respective platform (with a given version, ABI, etc.), whether they're built by CMake, autotools, meson or whatever. I can explain the reasons for this imperative requirement in more detail if desired, but it is utterly non-negotiable. I believe that conda-forge is not a "special" distribution in this regard - other distributions also have requirements to build things incrementally (both architecturally and practically), prefer shared libraries built once against a particular ABI, etc.

This has historically been extremely painful with bazel, to the point that many packaging efforts for packages natively built by bazel have either stalled completely (e.g. tensorflow-addons), are buggy (ray vs. protobuf interaction), or are maintained with extreme effort by a handful of people who know how to hack a bazel build enough to inject local dependencies (e.g. tensorflow).

For tensorflow in particular, this works by generating a custom toolchain, with lots of manual patching, and so on. It is unworkable at scale.

I was very much looking forward to the bzlmod effort reducing the difficulty of doing this, but now that I'm taking a closer look, AFAICT all of the documented overrides point to a repo, or at least need some bazel metadata (same for the given examples) This is very likely still an improvement over the previous state of affairs, but still leaves us with manually having to write that metadata on the fly for every consuming library, which is similarly unrealistic at the scale of 100s-1000s of packages.

Based on having an existing library with headers, library artefacts (either shared or static), and potentially some binaries (like protoc), what would help us[^1], in ascending order of awesomeness:

Note that this is not a new request, c.f. https://github.com/bazelbuild/bazel/issues/5252 or on SO (1, 2, ...). I've tried to look for related issues but didn't find anything for this point specifically.

[^1]: I only speak for myself; that said, I'm quite experienced in packaging, but I do not have the energy to deal with all the roadblocks that bazel builds put in my way. I think I can claim in good conscience that it would help many other people in conda-forge as well.

Which category does this issue belong to?

Configurability, External Dependency, Local Execution

What underlying problem are you trying to solve with this feature?

Distribute packages built by bazel, built against an existing (& well-controlled) set of prebuilt dependencies.

Which operating system are you running Bazel on?

Linux, MacOS, Windows

What is the output of bazel info release?

We'll generally build & use the last released bazel version

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

fmeum commented 10 months ago

Sounds very much related to https://github.com/bazelbuild/bazel/discussions/20581, although that's more about switching between Bazel and system version of given deps, which requires them to have been Bazelified in the first place.

I don't know much about conda, but maybe its packages could be turned into repos by a module extension? That is the most important new bit of Bzlmod that allows for better integration with package managers.