Open criemen opened 1 month ago
I guess the other question would be why we can't use the usual toolchain selection mechanism for the host toolchain, too, which naturally would eliminate the duplication.
I guess the other question would be why we can't use the usual toolchain selection mechanism for the host toolchain, too, which naturally would eliminate the duplication.
We use the toolchain in repository rules. There's a bootstrapping problem here - Bazel does the entire repository fetching phase before it allows accessing toolchains, because repository rules often generate toolchains.
When using
crate_universe
, the module extension unfortunately forces first a fetch of the host toolchain, then the module extension finishes, and only then will the target toolchain be fetched.That's in part due to the lazy-downloading of target toolchains (that's great!), in part because the crate_universe modext can't express multiple dependencies for preloading at the same time (as far as I'm aware, at least).
The result is that we're paying the (quite high) cost of unzipping the toolchain twice, as we can't parallelize this. This'd be less of a problem if the toolchain were
zstd
-compressed, but that doesn't look like it's happening anytime soon.What could we do? I don't know enough about what makes the host toolchain so special (with regards to the module extension attributes returned), so this is a bit speculative. Couldn't we, during target toolchain setup, detect if there's a host toolchain configured with the same tools, edition and version (that ought to be fairly standard), and then, instead of generating a separate
tools
repository, re-use the host toolchain? This might require moving therust_host_tools
modext out ofextensions.bzl
, which, per the documentation, is recommended anyway.Another way would be to offer a possibility of somehow force fetching the rust toolchain. That's quite tricky, as it needs to happen in parallel to
crate_universe
.For now, the best I could come up with was the following repository rule, but it is suuuuuper hacky and relies on a bunch of bazel and rules_rust implementation details.
This also then requires
in a/the build files where you build your rust code to trigger the toolchain pre-loading.