Support the Remote Execution API

[alercah]

My previous job at Google involved development of remote build execution for Bazel, and in the process we developed a standard API for it. Adding support to sccache for the Remote Execution API would allow it to build against a number of different servers with different characteristics, including a hosted GCP service currently in Alpha. There's also design underway for features like cross-compiles which would probably be good to avoid reinventing.

The architecture is a bit different from the architecture sccache currently uses for distributed builds, so it looks like it isn't a quick and easy changeover. The main difference appears to be that the sccache protocol seems (based on a quick skim; I might be wrong) to have the client interact directly with the build worker; the REAPI hides this and has a single unified frontend with the server handling task assignment. There is a worker API which is being worked on as well (it's not currently in the remote-apis repository but it is planned to be), which can be used to make the worker code more independent from the scheduler as well.

[luser]

Cool, thanks for all the useful info! I think this would be great to support. @aidanhs did all of the work to implement distributed compilation in sccache so he might have some thoughts to add here. The "client talks directly to build worker" architecture was implemented to mimic icecream AFAIK, with the expectation that transferring build artifacts around would be the slowest part of the whole operation and so we should try to avoid having an extra network hop there.

There's also design underway for features like cross-compiles which would probably be good to avoid reinventing.

It would be great if we could chat with some Remote Execution folks to figure out a way to make that work well, as distributed cross-compiles are a huge selling point.

Having Remote Execution support in sccache would be a really nice win for folks who don't have the resources to rewrite their entire build to use Bazel but want the ability to throw some money at making builds faster.

My colleague @gittup might also have thoughts here, he attended the most recent BazelConf.

[luser]

(Edited your comment to fix a broken link.)

[alercah]

I was one of the principal authors of the API, and I can get you in touch with my colleagues who are working on it currently. :)

For cross-compiles, the main area of work is standardizing the Platform specification so that it's easy to request specific platforms in a build. Of course, setting up other parts of the workers (e.g. available system libraries) will be important. If someone wants to take this on specifically, I'd definitely be happy to make an introduction.

As for sending artifacts directly back from the workers, it's not something we'd considered in earnest. It's true that it might be a bit faster, but it does mean that artifacts then need to get streamed separately to the cache, which would increase complexity significantly I think. A more fruitful area of work for Bazel specifically is trying to avoid fetching intermediate artifacts altogether until they are necessary or until a build is complete; I'm not sure it's viable in the current model because of how rustc --emit_info deps is used to get dependency information and that requires all the libraries to be present. I don't know enough about the dependency models to know if it might be possible to avoid requiring copies of the rlibs to work out all the compilation dependencies, and such a model probably would require deeper integration with Cargo as well.

If someone wants to pick this up (and possibly attend monthly video conference meetings where the API is discussed) I'd be happy to make an introduction to my former team. :)

[luser]

As for sending artifacts directly back from the workers, it's not something we'd considered in earnest. It's true that it might be a bit faster, but it does mean that artifacts then need to get streamed separately to the cache, which would increase complexity significantly I think. A more fruitful area of work for Bazel specifically is trying to avoid fetching intermediate artifacts altogether until they are necessary or until a build is complete

Ah, that makes sense given Bazel's model! Unfortunately with sccache we're acting as a simple compiler wrapper without full knowledge of the build graph so we can't do things quite as nicely as Bazel can. Even cargo can't quite do this because build scripts can run arbitrary code and they throw everything for a loop.

In any event, I don't think that particular bit would need to be a deal-breaker for adding support to sccache--we could probably just make remote execution against the API talk only to the scheduler if that's what currently exists. If he has time I'd like to get feedback from Aidan about that, but I don't think anyone is going to dive into implementing this right off the bat anyway. :)

A good first step would be for someone to write a crate that wraps the remote execution API to get some of the low-level Rust integration drudgery out of the way, which would make plugging it in to sccache as a prototype a lot more straightforward.

[alercah]

Ah, that makes sense given Bazel's model! Unfortunately with sccache we're acting as a simple compiler wrapper without full knowledge of the build graph so we can't do things quite as nicely as Bazel can. Even cargo can't quite do this because build scripts can run arbitrary code and they throw everything for a loop.

It actually required/requires a massive refactor in Bazel (I don't know the current status). So while it's theoretically awesome the build systems definitely aren't there yet. :)

Good point about build scripts, but I think it might be worth considering whether we can get those to move to remote execution too. Not all scripts will be able to but we could definitely imagine a world where most scripts can so that they can cache across multiple machines too.

In any event, I don't think that particular bit would need to be a deal-breaker for adding support to sccache--we could probably just make remote execution against the API talk only to the scheduler if that's what currently exists. If he has time I'd like to get feedback from Aidan about that, but I don't think anyone is going to dive into implementing this right off the bat anyway. :)

Yeah, for sure! At a high level, the response from a successful execution is the same as a cache hit, and can be handled identically.

A good first step would be for someone to write a crate that wraps the remote execution API to get some of the low-level Rust integration drudgery out of the way, which would make plugging it in to sccache as a prototype a lot more straightforward.

That's a good idea!

[edbaunton]

You might be able to reuse some of the work we have been doing on recc which is supposed to be a replacement for ccache that uses Remote Execution and Caching. Currently that is all wrapped up into one but operates very similarly to ccache's depend mode (I think).

Doesn't that achieve the same thing as what sccache is trying to do but via the remote execution protocol as you proposed?

[alercah]

Yeah, it looks like the same idea. Either sccache adding support for REAPI or recc adding support for Rust would attack this use case.

[aidanhs]

The design of sccache distributed compile was motivated by the situation as described to me at Mozilla and in my own use-cases, where, IIRC:

• there are a bunch of powerful machines that aren't that busy a lot of the time, and could use their spare time doing compilations
• getting officially supported powerful hardware (big storage, big bandwidth) from IT is not necessarily easy/cheap

My reading of the remote execution protocol (particularly this bit about actions) indicates (if I'm reading it correctly) that the protocol requires the presence of a content-addressible storage that all actual data is communicated via (source file inputs, outputs etc).

This probably makes sense if you have an office-wide blessed build server with a high-performance CAS, but I'd need to think more about the performance in a situation where you have smaller office(s) with less centralised resource. The sccache scheduler is deliberately extremely 'light' in terms of resource requirements exactly because centralised resource is (IME) limited until you get to a certain office size (at which point you probably have a build tools team with a budget).

Honestly though, much of this is based on intuition and/or guesswork - I'd love to do some performance modelling to better understand what makes the difference here (and if you have any that would be amazing). It may be that round-trip time is irrelevant because typical individual compilations take minutes. I'd also be really interested to understand a 'best practice' setup for a build environment, particularly around ensuring that the action cache and CAS is performant.

In terms of implementing it in sccache (presumably just on the client side) - to comply with the spirit of the api, you would probably have a 'remote execution mode' for sccache since off the top of my head it changes the 'cache or compile' flow. It would be good to get a view of a 'best practice' architecture so the sccache docs can offer advice on when each setup makes sense (e.g. an s3 cache if your worker and client are in the same building doesn't make sense given the performance of s3 calls).

but it does mean that artifacts then need to get streamed separately to the cache, which would increase complexity significantly I think

Can you elaborate a bit on this? Sccache currently trusts the client to update the cache, and intuitively if we say reads are more common than writes then the extra hop isn't such a big deal. I could see a good argument for giving workers the ability to do this (e.g. if we trust workers more than clients). I'm not sure where the complexity comes from though.

[alercah]

The design of sccache distributed compile was motivated by the situation as described to me at Mozilla and in my own use-cases, where, IIRC:

• there are a bunch of powerful machines that aren't that busy a lot of the time, and could use their spare time doing compilations
• getting officially supported powerful hardware (big storage, big bandwidth) from IT is not necessarily easy/cheap

My reading of the remote execution protocol (particularly this bit about actions) indicates (if I'm reading it correctly) that the protocol requires the presence of a content-addressible storage that all actual data is communicated via (source file inputs, outputs etc).

This is mostly correct---the only thing not stored in the CAS is the action cache, since it's addressed by the hash of the action. But it only stores pointers into the CAS, not outputs.

This probably makes sense if you have an office-wide blessed build server with a high-performance CAS, but I'd need to think more about the performance in a situation where you have smaller office(s) with less centralised resource. The sccache scheduler is deliberately extremely 'light' in terms of resource requirements exactly because centralised resource is (IME) limited until you get to a certain office size (at which point you probably have a build tools team with a budget).

This is generally the assumption. When I originally filed this ticket, I was only envisioning that sccache might be used as a client to an existing server implementation, because for instance it already knows how to package up a Rust compilation for execution on another machine. In this case, I would assume that sccache wouldn't run a server component at all. It might be possible to make a version of the protocol where clients can register themselves as owning a copy of a piece of data, and having them serve CAS entries directly, or using a peer-to-peer CAS (I don't actually know what sccache does), but I think that's a bigger-scope question.

Honestly though, much of this is based on intuition and/or guesswork - I'd love to do some performance modelling to better understand what makes the difference here (and if you have any that would be amazing). It may be that round-trip time is irrelevant because typical individual compilations take minutes. I'd also be really interested to understand a 'best practice' setup for a build environment, particularly around ensuring that the action cache and CAS is performant.

I think it really highly depends on the topology you have and the distribution of resources. If the available workers are regularly saturated, for instance, then extra round-trip time to queue jobs by putting data in the CAS is not likely to be an issue unless you also saturate the available bandwidth of the CAS (which in large-scale implementations I would expect to be distributed so that this is not a concern). I did some analysis of a different question here which may nonetheless be a helpful starting point.

In terms of implementing it in sccache (presumably just on the client side) - to comply with the spirit of the api, you would probably have a 'remote execution mode' for sccache since off the top of my head it changes the 'cache or compile' flow. It would be good to get a view of a 'best practice' architecture so the sccache docs can offer advice on when each setup makes sense (e.g. an s3 cache if your worker and client are in the same building doesn't make sense given the performance of s3 calls).

Yes, this makes sense to me.

but it does mean that artifacts then need to get streamed separately to the cache, which would increase complexity significantly I think

Can you elaborate a bit on this? Sccache currently trusts the client to update the cache, and intuitively if we say reads are more common than writes then the extra hop isn't such a big deal. I could see a good argument for giving workers the ability to do this (e.g. if we trust workers more than clients). I'm not sure where the complexity comes from though.

On reread, I'm not sure either XD. Though it's worth noting that if you take a peek at the doc I linked above, it was about a 2:1 ratio of new actions to cached ones in the usage I was analyzing, though I can't say that's universal (particularly across different languages). The primary client was Bazel, which also does local caching and therefore the server will not see any results that are already kept locally. I would imagine that the actual ratio would depend significantly on the profile of the work being done: someone working on testing compiler optimizations on a large project may have very few cache hits, while a CI system building from scratch every time (and in particular discarding the build system's local cache) may be dominated by them.

[luser]

FYI I learned recently that the pants build tool (Twitter's bazel-alike) is partially implemented in Rust and they have an implementation of the bazel remote execution API: https://github.com/pantsbuild/pants/blob/2f373c1f8ad66975578ddc676217353d12908749/src/rust/engine/process_execution/src/remote.rs#L228

They'd likely be amenable to having that factored out into a reusable crate.