search

github / codeql-action

Actions for running CodeQL analysis
MIT License
1.14k stars 313 forks source link

Compress release bundle with zstandard/zstd to reduce size #2400

Open DSmithVA opened 1 month ago

DSmithVA commented 1 month ago

I propose a .zstd download option alongside the existing .gz one for Linux releases. For the latest 2.18.1 linux64 bundle, using zstd instead of gzip can cut off 33% of the file size, or 822.8 MiB down to 553.2 MiB.

Example command to convert the existing .gz: zcat codeql-bundle-linux64.tar.gz | zstd --long=27 -9 -o codeql-bundle-linux64.tar.zstd

File sizes: 862823301 codeql-bundle-linux64.tar.gz 580124258 codeql-bundle-linux64.tar.zstd

For zstd arguments, compression levels above -9 saw diminishing returns, though -19 does get down to 504.5 MiB while taking 12x longer to compress. Using higher --long= values improves compression, but 27 is the highest value that clients can process by default, per https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md?plain=1#L162

Compression with xz is also an improvement, it's just noticeably slower. Either is an improvement over just .gz and any recent linux will support both .zstd or .xz for decompression.

jketema commented 1 month ago

Thanks for your feedback. We'll take this into consideration.