bitrise-steplib / bitrise-step-save-cache

1 stars 4 forks source link

Fine tune compression params #10

Closed ofalvai closed 1 year ago

ofalvai commented 1 year ago

Checklist

Version

Requires no version update

Context

Fine-tuning zstd's compression parameters for our use case.

Changes

Disable long-distance matching. Based on my benchmarks, long-distance matching improves compression ratio very little, but slows down compression significantly (especially on Linux)

Investigation details

I made the benchmarks on Bitrise VMs using zstd's benchmark mode and a tar of a typical Gradle build cache.

Commands:

tar -cf test.tar ~/.gradle/caches ~/.gradle/wrapper

zstd -b3 -T0 test.tar

zstd -b3 -T0 --long test.tar

Results:

Numbers are: compression ratio, compression speed, decompression speed

Linux (Standard VM):

1.4.4:
-b3 -T0: (1.921), 551.4 MB/s ,1595.3 MB/s
-b3 -T0 --long: (2.089), 257.3 MB/s ,1705.2 MB/s

1.5.3:
-b3 -T0: (x1.920),  576.3 MB/s, 1788.7 MB/s
-b3 -T0 --long: (x2.084),  414.3 MB/s, 1943.3 MB/s

Linux uses an older version of zstd, the latest version has some speed gains, but --long is still a lot slower.

macOS:

macOS Intel (Elite):
1.5.2:
-b3 -T0: (x1.955), 1078.7 MB/s, 1815.8 MB/s
-b3 -T0 --long: (x2.111),  927.3 MB/s, 1880.2 MB/s

macOS M1 (EliteXL):
1.5.2:
-b3 -T0: (x1.955), 2406.7 MB/s, 2237.9 MB/s
-b3 -T0 --long: (x2.111),  909.2 MB/s, 2389.0 MB/s

Decisions