search

Blosc / c-blosc

A blocking, shuffling and loss-less compression library that can be faster than `memcpy()`.
https://www.blosc.org
Other
983 stars 158 forks source link

Add optional support for zlib-ng #132

Open jack-pappas opened 9 years ago

jack-pappas commented 9 years ago

zlib-ng is a fork of zlib which is working to "modernize" the zlib codebase. It includes some patches from Intel and Cloudflare which use the SSE 4.2 pclmulqdq instruction to optimize the CRC32 calculations for compressed blocks, leading to a significant speedup for both compression and decompression.

It seems like a promising project but it'd be important to perform in-depth testing before trusting any important data to it. Perhaps a compromise would be to import zlib-ng to this repository (or add it as a submodule so it's easier to stay up-to-date), then implement a CMake option to use it instead of the standard zlib when compiling blosc to allow for experimentation and verification.

FrancescAlted commented 9 years ago

Yes, zlib-ng looks like a perfect match for Blosc. I agree with allowing a transition time where zlib-ng and classic zlib would coexist with a CMake option to allow to switch to zlib-ng. And provided that zlib-ng is still in-flux, I think it is a good idea to add it as a submodule.