Adding feature to compress cache content

[benjaminfuchs]

Pull request for issue #327

I tried to implement the feature similar to ccache. It can be controlled by the following two env variables:

CLCACHE_COMPRESS If true, clcache will compress object files it puts in the cache. If the cache was filled without compression it can't be used with compression and vice versa (i.e. you have to clear the cache when changing this setting). The default is false.

CLCACHE_COMPRESSLEVEL This setting determines the level at which clcache will compress object files. It only has effect if compression is enabled. The value defaults to 6, and must be no lower than 1 (fastest, worst compression) and no higher than 9 (slowest, best compression).

[codecov-io]

Codecov Report

Merging #328 into master will increase coverage by 0.13%. The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #328      +/-   ##
==========================================
+ Coverage   88.23%   88.37%   +0.13%     
==========================================
  Files           4        4              
  Lines        1275     1290      +15     
  Branches      190      194       +4     
==========================================
+ Hits         1125     1140      +15     
  Misses        111      111              
  Partials       39       39

Flag	Coverage Δ
#integrationtests_memcached	67.09% <82.6%> (+0.23%)	:arrow_up:
#unittests	85.27% <91.3%> (+0.01%)	:arrow_up:

Impacted Files	Coverage Δ
clcache/__main__.py	90.25% <100%> (+0.13%)	:arrow_up:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 6d62cc1...624a2c9. Read the comment docs.

[benjaminfuchs]

@frerich We are now using this dev version and it works stable in our env.

[frerich]

@benjaminfuchs This looks interesting! Did you ever profile the effect of compressing the cache contents?

My gut reaction was that compressing cached files might be counter-productive, since it requires extra CPU time. However, maybe smaller files improve the disk cache hit rate, which may outweigh the time needed for (de-)compressing data?

Or is the main purpose maybe not to accelerate anything at all but rather to reduce the disk usage of the cache?

[benjaminfuchs]

@frerich Thanks for the feedback!

In our setup the cache size goes up to multiple GBs and on our build servers IO throughput got the bottleneck. I wouldn't say the compression is accelerating the build but it's also not slowing it down.

We also use ccache with compression on. This way we can have multiple cache folders side by side for different builds and save disk space.

It hardly depends on the use case if compression makes sense and in our case it helps.

[frerich]

Looks good to me, let's give it a try. Thanks!

[pitrou]

I'm gonna try this on our AppVeyor builds. As a sidenote, it would be nice to make the compression scheme configurable. Using something like zstd should give higher compression ratios at the same (de)compression speed, or much larger speeds at similar compression ratio.