search

freebsd / pkg

Package management tool for FreeBSD. Help at #pkg on Libera Chat or pkg@FreeBSD.org
Other
737 stars 277 forks source link

FreeBSD 13.1 is EOL, switch to zstd? #2277

Open cgull opened 2 months ago

cgull commented 2 months ago

FreeBSD 13.1 has been EOL for 8 months. Is it time to switch the default compression from xz to zstd on 13.x yet?

FWIW, I've been recompressing all FreeBSD dist tarballs, and all packages except pkg itself, to zstd for 13.x for a couple of years on 13.x, with no problems that I can recall.

rilysh commented 2 months ago

In a quick look, zstd seems significantly faster in compression and decompression (almost) whereas xz is significantly slower in compression but on decompression it's almost as fast as zstd. The only (probably ignorable) downside seems to is that zstd compressed result is quite larger than the xz compressed result.

cgull commented 2 months ago

You need to look deeper. pkg and FreeBSD have already switched to zstd for 14.0-RELEASE.

zstd's default compression parameters produce worse compression than xz (but it's very fast). But it is not hard to select compression parameters that get similar compression times and smaller file sizes than xz. zstd decompression is always much faster than xz (around 10x), and compression parameters don't change decompression performance much.

Try zstd --19 --long, for example.

I haven't looked at the choices made for FreeBSD 14 and pkg.

rilysh commented 2 months ago

I don't think overriding the default will yield anything significant for data that can't be compressed any further. Disk space isn't a vital thing nowadays so in most cases the default compression is just fine.

For decompression, it's not exactly 10x faster, and mostly depends on the archive size that it's trying to decompress. For smaller archives xz decompression speed is similar to zstd.