ornladios / ADIOS2

Next generation of ADIOS developed in the Exascale Computing Program
https://adios2.readthedocs.io/en/latest/index.html
Apache License 2.0
271 stars 126 forks source link

Add a ZSTD compression operator #3948

Open vicentebolea opened 11 months ago

vicentebolea commented 11 months ago

ZSTD is an industry standard lossless parallel compressor brought by Facebook that promises a strong compression ratio with an high throughput. It also provides the possibility of generating and using a dictionary to increase small files (de)compression, this can be a key feature. https://facebook.github.io/zstd/.

I believe it will be beneficial to replace the BZIP2 compressor to the zstd for better performance overall.

Here is a benchmark performed by CentmintMod

Screenshot from 2023-11-28 19-03-01

https://community.centminmod.com/threads/round-3-compression-comparison-benchmarks-zstd-vs-brotli-vs-pigz-vs-bzip2-vs-xz-etc.17259/

williamfgc commented 11 months ago

Looks neat, could be an optional dependency that coexists with bzip2. Operators in general weren't designed to compete, but rather driven by user requests.