Open firebird-automations opened 4 years ago
Commented by: @dyemanov
For record-level (de)compression I'd rather consider LZ4 instead.
Commented by: @AlexPeshkoff
WireCompression makes sense for slow links (like modem) in WAN case. For LAN with gigabit link it does not make sense. May be in WAN case with fast links zstandard will be efficient. BTW, does format of compressed stream differ for zlib/zstd?
Commented by: Basil A. Sidorov (basid)
Customers may use LAN-WAN combination: EDS between branch office and, typically WireCompression are globally enabled in firebird.conf zstd packet format, of course, are different from deflate packet format.
Commented by: Sascha Michel (datiscum1)
I've been watching the zlib-ng project for a while and did some speed tests today. Of course, it's not a leap like zstd, but it's clearly faster and above all 100% compatible with the old zlib. The Intel library "zpipe_ipp" already caused me compatibility problems when I wanted to unpack data with a standard zlib. Maybe someone will think about the zlib-ng for Firebird.
https://github.com/zlib-ng/zlib-ng Zlib-ng can be compiled in zlib-compat mode, suitable for zlib-replacement in a single application or system-wide.
Please note that zlib-ng in zlib-compat mode is API-compatible but not ABI-compatible, meaning that you cannot simply replace the zlib library/dll files and expect the application to work. The application will need to be recompiled against the zlib-ng headers and libs to ensure full compatability.
zpipe_libzOrg = static linked with standard zlib. zpipe = static linked with zlib-ng compat NOT optimized for system cpu zpipe_SystemOptimized = static linked with zlib-ng compat optimized for system cpu
zpipe_ipp "Building a faster ZLIB with Intel Integrated Performance Primitives"
Database does not contain packed data.
test@HillSrv:/DB$ mbuffer -i JT_EUR_3Uhr.fbk | zpipe_libzOrg | mbuffer -o /dev/null summary: 428 MiByte in 5.1sec - average of 84.3 MiB/s summary: 93.1 MiByte
test@HillSrv:/DB$ mbuffer -i JT_EUR_3Uhr.fbk | zpipe | mbuffer -o /dev/null summary: 428 MiByte in 2.9sec - average of 146 MiB/s summary: 90.6 MiByte
test@HillSrv:/DB$ mbuffer -i JT_EUR_3Uhr.fbk | zpipe_SystemOptimized | mbuffer -o /dev/null summary: 428 MiByte in 2.7sec - average of 161 MiB/s summary: 90.6 MiByte
---
test@HillSrv:/DB$ mbuffer -i JT_EUR_3Uhr.fbk | zstd | mbuffer -o /dev/null summary: 428 MiByte in 1.3sec - average of 339 MiB/s summary: 71.2 MiByte
test@HillSrv:/DB$ mbuffer -i JT_EUR_3Uhr.fbk | lz4 -3 | mbuffer -o /dev/null summary: 428 MiByte in 3.6sec - average of 119 MiB/s summary: 88.9 MiByte
test@HillSrv:/DB$ mbuffer -i JT_EUR_3Uhr.fbk | zpipe_ipp | mbuffer -o /dev/null summary: 428 MiByte in 2.4sec - average of 182 MiB/s summary: 94.6 MiByte
Database contains 99% blobs packed with zlib.
test@HillSrv:/DB$ time mbuffer -i MATWDE_20180815.fbk | zpipe_libzOrg | mbuffer -o /dev/null summary: 1187 MiByte in 27.7sec - average of 42.8 MiB/s summary: 1177 MiByte
test@HillSrv:/DB$ mbuffer -i MATWDE_20180815.fbk | zpipe | mbuffer -o /dev/null summary: 1187 MiByte in 20.0sec - average of 59.4 MiB/s summary: 1177 MiByte
test@HillSrv:/DB$ mbuffer -i MATWDE_20180815.fbk | zpipe_SystemOptimized | mbuffer -o /dev/null summary: 1187 MiByte in 19.5sec - average of 60.7 MiB/s summary: 1177 MiByte
---
test@HillSrv:/DB$ mbuffer -i MATWDE_20180815.fbk | zstd | mbuffer -o /dev/null summary: 1187 MiByte in 1.4sec - average of 843 MiB/s summary: 1180 MiByte
test@HillSrv:/DB$ mbuffer -i MATWDE_20180815.fbk | lz4 | mbuffer -o /dev/null summary: 1187 MiByte in 2.0sec - average of 583 MiB/s summary: 1183 MiByte
test@HillSrv:/DB$ mbuffer -i MATWDE_20180815.fbk | zpipe_ipp | mbuffer -o /dev/null summary: 1187 MiByte in 21.7sec - average of 54.6 MiB/s summary: 1177 MiByte
Submitted by: Basil A. Sidorov (basid)
Attachments: README-zlibWrapper.md
Votes: 1
http://zstandard.net - stable and mature project, offered fast (de)compressor with many features and dual (BSD/GPL) licensing. Firebird 3.0 may use WireCompression, but zlib are slow for modern gigabit link. Fast (de)compression may be profitable for record chains in database file, for database crypt and so on.
For quick transition from zlib to zstd exist zlibwrapper as part of zstd source tree.