Support zstd/lz4/zchunk as compression method for ostree repos

[travier]

Description

During rpm-ostree compose, a lot of time is spent in compressing files for inclusion in an ostree repo. The currently available compression algorithm in ostree is single threaded zlib. Adding support for better / multi-threaded compression algorithms could reduce the bandwidth used and reduce compose times.

Options:

• lz4: fast
• zstd: good compromise
• zchunk: could even allow potentially large files in an ostree repo to be split up and thus further reduce the size of updates that only change parts of a file.

Reference

See: https://discussion.fedoraproject.org/t/kinoite-a-kde-and-now-xfce-version-of-fedora-silverblue/147/2

[cgwalters]

I think it would make sense to introduce archive-zstd or so.

During rpm-ostree compose, a lot of time is spent in compressing files for inclusion in an ostree repo.

This however isn't required - ostree has uncompressed modes, and to optimize the compression part the best approach is to have the previous commit pulled; ostree will then already avoid compressing already extant objects. It's more just that in the current coreos-assembler flow we don't use any shared caches, which would fix this. But bigger picture there we're moving to the ostree-native container flow which will be the canonical delivery format, and that's already compressed. (OCI defaults to gzip, but as of recently supports zstd too)