FirebirdSQL / firebird

Firebird server, client and tools
https://www.firebirdsql.org/
1.23k stars 213 forks source link

Change zlib level compression from "best" to fast #8174

Open basid-irk opened 2 months ago

basid-irk commented 2 months ago

Change zlib deflate compression level from Z_DEFAULT_COMPRESSION (6 for now) to Z_BEST_SPEED (1).

int ret = zlib().deflateInit(&port_send_stream, Z_DEFAULT_COMPRESSION);  // remote/remote.cpp
int ret = zlib().deflateInit(&strm, Z_DEFAULT_COMPRESSION);              // burp/mvol.cpp

As can see on comparision table lz4 project, deflate Z_DEFAULT_COMPRESSION are "little slow" (about 36 MB/s). Deflate Z_BEST_SPEED (1) are almost three times faster (about 100 MB/s) with very small loss on compression degree and decompression speed for "typical" data and will much faster on incompressible data.

TommiPrami commented 2 months ago

Compression algorithm and compression level maybe should be configurable.

ZLib for backwards support, LZ4 and Zstd might be the best choices. It all depends on everything. Data and transfer link speed (to/from hdd/ssd/super fast Ethernet/fast ethernet/slow internet etc...).

There most likely is use case when even very slow compression speed but good compression ratio of lzma would be best choice. Way back I used lzma with full compression level to make data transfer faster over very slow gprs transfer... Because transfer was so slow...

But ZLib/LZ4/Zstd for choices and compression level most likely would be more than enough.

-Tee-

Benchmarks etc:

https://indico.fnal.gov/event/16264/contributions/36466/attachments/22610/28037/Zstd__LZ4.pdf https://manishrjain.com/compression-algo-moving-data