search

mozilla / sccache

Sccache is a ccache-like tool. It is used as a compiler wrapper and avoids compilation when possible. Sccache has the capability to utilize caching in remote storage environments, including various cloud storage options, or alternatively, in local storage.
Apache License 2.0
5.76k stars 545 forks source link

[feature] Multiple cache levels #1020

Open browdus opened 3 years ago

browdus commented 3 years ago

Hi everyone! Is it possible, or has it been considered, to allow multiple levels of cache? For instance, using local disk as an L1 cache, and s3/azure/gcs as L2.

I envision it working something like the following:

Check disk cache according to current rules
if (found)
    return item

check s3/azure/gcs according to current rules
if (found)
    return item

compile item
populate disk cache
populate s3/azure/gcs cache

return item

This lets us use nvme as a hot cache, while still being able to hit the network cache if needed. This (theoretically) avoids a lot of network traffic and associated costs, and reduces build times on average. It also gives us a transparent way to seed new build hosts when they spin up, which makes automatic scaling of the build fleet more effective. Since each host has it's own hot cache, the load on s3/azure/gcs should scale better than linearly with the number of build hosts (on average; linear being the worst case).

Generically, it could be useful to mix/match all the cache types. For example, local disk as L1, on-prem memcached/redis as L2, and s3/azure/gcs as L3. Such a setup could be used to share build cache with all the devs in an office without hitting the internet as much, while still being able to share that build cache with other offices, etc.

Thanks!

browdus commented 3 years ago

related: https://github.com/mozilla/sccache/pull/241, https://github.com/electron/electron/issues/21006, https://github.com/mozilla/sccache/issues/30

browdus commented 3 years ago

For anyone wanting to do something similar, you can hack something together by combining ccache and sccache. The result is not optimal (multiple compression steps, for example), but it works reasonably well. This effectively gives you an L1 disk cache, and an L2 of anything sccache can handle.

export CCACHE_PREFIX="sccache"
ccache gcc ...

If you are using cmake, make sure you don't have something like the following, as sccache will fail to handle linking (as designed).

set_property(GLOBAL PROPERTY RULE_LAUNCH_LINK ccache)
Be-ing commented 3 years ago

set_property(GLOBAL PROPERTY RULE_LAUNCH_LINK ccache)

Use CMAKE_C_COMPILER_LAUNCHER / CMAKE_CXX_COMPILER_LAUNCHER rather than RULE_LAUNCH_COMPILE: https://github.com/mozilla/sccache/issues/947