Open basid-irk opened 2 months ago
Compression algorithm and compression level maybe should be configurable.
ZLib for backwards support, LZ4 and Zstd might be the best choices. It all depends on everything. Data and transfer link speed (to/from hdd/ssd/super fast Ethernet/fast ethernet/slow internet etc...).
There most likely is use case when even very slow compression speed but good compression ratio of lzma would be best choice. Way back I used lzma with full compression level to make data transfer faster over very slow gprs transfer... Because transfer was so slow...
But ZLib/LZ4/Zstd for choices and compression level most likely would be more than enough.
-Tee-
Benchmarks etc:
https://indico.fnal.gov/event/16264/contributions/36466/attachments/22610/28037/Zstd__LZ4.pdf https://manishrjain.com/compression-algo-moving-data
Change zlib deflate compression level from Z_DEFAULT_COMPRESSION (6 for now) to Z_BEST_SPEED (1).
As can see on comparision table lz4 project, deflate Z_DEFAULT_COMPRESSION are "little slow" (about 36 MB/s). Deflate Z_BEST_SPEED (1) are almost three times faster (about 100 MB/s) with very small loss on compression degree and decompression speed for "typical" data and will much faster on incompressible data.