Open JackKelly opened 4 years ago
+1 that's great analysis!
Also consider turning of Blosc bit shuffle for another performance boost (at the cost of some compression size).
There are more benchmarks here: http://alimanfoo.github.io/2016/09/21/genotype-compression-benchmark.html
As per @tv3141's work in https://github.com/tv3141/metoffice_ec2/blob/storage_optimisation_notebook/storage_optimisation/optimize_storage.ipynb
zstd is considerably faster to compress.