REAPI (Google/Bazel Remote Execution API) Support

[kinnison]

Building large Rust projects can often be done in parallel thanks to the large dependency trees which tend to result from any non-trivial project. It would be nice to be able to parallelise this onto multiple systems. In the C/C++ world, one might use distcc to do this, in the modern(?) world this is often done via a mechanism such as REAPI.

Describe the solution you'd like

It would be wonderful if Cargo were able to be configured to use an REAPI service to run builds. Obviously the REAPI service would have to provide a platform with a compatible compiler but given that existing, it ought to be possible for common builds to parallelise in this way.

Notes

REAPI requires that the full set of inputs be understood for sending to the REAPI executor. This can be naïvely done by sending the entire outer crate's contents including target contents each time, along with the command needing running. These wouldn't be too complex to construct without even downloading the intermediates from the REAPI service which should minimise the cost of constructing such maximal jobs.

I have some experience of REAPI through my job and would be pleased to discuss this further, indeed if it might be possible to prototype this as a RUSTC_WRAPPER then I'd be up for discussing writing such, though I feel it'll need to be in cargo proper eventually because it affects the number of parallel jobs you could be running.

[alexcrichton]

An interesting idea! We have a primitive concept of an executor which could be the beginnings of this, but having built-in support would be interesting as well! That being said though I suspect that this would require some thoughtful design to get the most out of it, and may need upstream rustc changes about how crates move around and such.

[kinnison]

I'm glad to hear there's interest at least. I think there's lots of careful design work going to be needed to get something performant out of it, but it's worth thinking about at least briefly in the near-term. I wonder if perhaps it might be a good topic for the next allhands, where we could get a bunch of cargo and compiler people together to discuss what'd be needed to make this possible / useful. It'd also potentially improve matters for integration of Rust code into a Bazel build environment.

[luser]

FYI I learned recently that the pants build tool (Twitter's bazel-alike) is partially implemented in Rust and they have an implementation of the bazel remote execution API: https://github.com/pantsbuild/pants/blob/2f373c1f8ad66975578ddc676217353d12908749/src/rust/engine/process_execution/src/remote.rs#L228

They'd likely be amenable to having that factored out into a reusable crate.

[kinnison]

@luser Having a nice crate which is a reusable CAS/ByteStream/ExecutionService client and so on would be super-useful. I have considered writing one myself in the past, but if refactoring one out of pantsbuild would be faster, then that'd make an awful lot of sense.

[luser]

In the meantime you can look at sccache's distributed build support if you want something that works right now. It's similar to icecream in that it can package up your local toolchain for remote execution, but we had it built with security features from the start so it has transport-layer encryption and sandboxed execution of compiles.

[kinnison]

@luser Oh interesting, perhaps the answer is to add REAPI to sccache instead?

[luser]

A developer who had worked on the REAPI opened an issue for exactly that! https://github.com/mozilla/sccache/issues/358

[allada]

For anyone interested in picking this up at some point, here's a project I wrote that is a rust-implementation of the Bazel Remote Execution API (both CAS and execution capabilities): https://github.com/allada/turbo-cache

It might be useful to rip out any libraries, tooling or just to get a sample of a working server-side implementation.