search

awalsh128 / cache-apt-pkgs-action

Cache APT packages in GitHub Actions
Other
205 stars 35 forks source link

Improved package files compression with zstd. #46

Closed awalsh128 closed 2 years ago

awalsh128 commented 2 years ago

Mentioned in issue #45 by @tamascsaba

Other improvement what I suggest is use zstd compression. It is much faster and supported in hosted runners.: https://github.com/actions/virtual-environments/issues/89 You can use with: tar -caf package.tar.zst package-directory

Will perform some due diligence and update this when I get a chance.

awalsh128 commented 2 years ago

Looks like the main citing of this performance is from FB engineering (who designed it) which looks impressive. https://engineering.fb.com/2016/08/31/core-data/smaller-and-faster-data-compression-with-zstandard The tested corpus matches our use case too (binaries < 51MB).

Just curious, have you observed this performance increase in the wild too @tamascsaba?

tamascsaba commented 2 years ago

Yes the decompress speed of zstd is amazing 👍

image

I checked and you use action/cache and maybe it is worth it to compress with zstd or gzip or anything else, because action/cache compress with zstd by default and you will compress twice.

awalsh128 commented 2 years ago

Ah good point. I think you mean to not compress?

tamascsaba commented 2 years ago

Taking a look at the performance differences is worth it in my opinion. For example try to install a bigger package and cache without compress or only use tar or compress with zstd or compress with gzip.

awalsh128 commented 2 years ago

Was thinking this over and may omit compression entirely since the cache action already performs compression. This would reduce the cycles needed to backup the files too. May run a benchmark to test this hypothesis against a fairly large package.

awalsh128 commented 2 years ago

Did some testing and concluded it would be faster to remove compression altogether, especially considering that the actions/cache already does this.