openclimatefix / metoffice_ec2

Subset Met Office MOGREPS-UK and UKV on AWS EC2
MIT License
11 stars 2 forks source link

Use zstd instead of LZMA #21

Open JackKelly opened 4 years ago

JackKelly commented 4 years ago

As per @tv3141's work in https://github.com/tv3141/metoffice_ec2/blob/storage_optimisation_notebook/storage_optimisation/optimize_storage.ipynb

zstd is considerably faster to compress.

tomwhite commented 4 years ago

+1 that's great analysis!

Also consider turning of Blosc bit shuffle for another performance boost (at the cost of some compression size).

There are more benchmarks here: http://alimanfoo.github.io/2016/09/21/genotype-compression-benchmark.html