search

mozilla / sccache

Sccache is a ccache-like tool. It is used as a compiler wrapper and avoids compilation when possible. Sccache has the capability to utilize caching in remote storage environments, including various cloud storage options, or alternatively, in local storage.
Apache License 2.0
5.68k stars 539 forks source link

new compression format for cached artifacts #759

Open froydnj opened 4 years ago

froydnj commented 4 years ago

Zip files are super-convenient, but maybe using something that would provide a better compression/performance ratio would be better. At least in the common case for C/C++ compiles where you have a single output and no compiler messages. Maybe you could invent a small custom format for the typical Rust compile (three outputs) as well.

luser commented 4 years ago

Related: https://github.com/mozilla/sccache/issues/552 .

IIRC the zip file cache entry format dates back to the original Python version of sccache. I don't know that inventing a custom format is worthwhile, but just switching to zstd-compressed tarballs ought to be a pretty simple change.

mostynb commented 4 years ago

The bazel remote execution api's ActionResult protobuf message might be worth considering over tarballs. It provides a bit more structure, and is flexible enough to handle multiple output files (which would be good to support for C/C++ compiles too). It would also make it easier to integrate with REAPI cache servers.

https://github.com/bazelbuild/remote-apis/blob/7802003e00901b4e740fe0ebec1243c221e02ae2/build/bazel/remote/execution/v2/remote_execution.proto#L854