search

Raku / whateverable

🤖 Different IRC bots that operate on a bunch of prebuilt Rakudo versions
https://gist.github.com/Whateverable
GNU Affero General Public License v3.0
18 stars 14 forks source link

Replace lrzip with zstd #389

Open AlexDaniel opened 9 months ago

AlexDaniel commented 9 months ago

See https://github.com/Raku/whateverable/issues/23#issuecomment-670971601 for some context. When lrzip was introduced, zstd did not have a long-range mode, so lrzip was the only choice to achieve what we needed. Today the situation is different, and we can use zstd for everything.

AlexDaniel commented 9 months ago

Actually, because of https://github.com/facebook/zstd/issues/3062, it's not a win-win situation.

For 20 rakudo builds the results are like this: Command Compression time File size Decompression time
lrzip -q -L 9 ≈26s ≈8M ≈5.3s
zstd -19 --long=31 ≈97s ≈12M ≈0.8s
zstd --long=31 ≈3.7s ≈16M ≈0.8s

It's amazing what zstd can do at just double the size. However, CPU cycles are cheap, so I don't mind if the server uses the idle time to achieve a slightly higher compression rate. That said, I wish zstd was able to reach lrzip's compression ratio.