search

mbhall88 / compression_benchmark

Benchmarking FASTQ compression with 'mature' compression algorithms
MIT License
30 stars 4 forks source link

Add LZ4 #8

Closed mbhall88 closed 1 week ago

mbhall88 commented 2 weeks ago

Have you looked into lz4 (https://github.com/lz4/lz4) and LZHAM (https://github.com/richgel999/lzham_codec)? While lz4 will achieve compression ratios inline with the lower levels of gzip or just below, it will also achieve decompression speeds in the GB/s range. Usually within the same order of magnitude of memory copy speed (https://github.com/lz4/lz4/tree/dev?tab=readme-ov-file#benchmarks).

_Originally posted by @lpsantil in https://github.com/mbhall88/compression_benchmark/issues/1#issuecomment-2350788060_

mbhall88 commented 1 week ago

Added

lpsantil commented 1 week ago

What are your first impressions? There is the one graph where lz4 has slightly higher decompression rates. Memory usage seems to be very low as well.

mbhall88 commented 1 week ago

It's fast. But compression rates aren't great. I think zstd still strikes the best balance of speed and compression ratios in terms of compression algorithms, but uBAM and uCRAM are definitely superior if we can start getting more tools to accept those as input/output sources.