iclab / centinel

http://iclab.org/
MIT License
34 stars 17 forks source link

Replace bzip2 with zstd or lz4 #296

Open zackw opened 5 years ago

zackw commented 5 years ago

bzip2 compression produces more compact files on disk than gzip, but it's also very slow both to compress and to decompress. We should consider switching to a newer compression algorithm. Fast decompression should be a priority. zstd is somewhat faster than bzip2 (especially at decompression) for equivalent compressed sizes, and lz4 is much faster than bzip2 but only about as good as gzip when it comes to compressed sizes.