search

ACDguide / Governance

Main repository for Climate data guidelines working group
https://acdguide.github.io/Governance
Other
3 stars 3 forks source link

zarr and object store like formats as an alternative to netcdf #91

Open paolap opened 1 month ago

paolap commented 1 month ago

As this is a hot topic in the community it would be nice to have a page somewhere where pros and cons and implications are discussed. It's important for people who managed data to have a t east a starting point to look into this when taking decision around formats

see https://docs.unidata.ucar.edu/netcdf-c/current/md__media_psf_Home_Desktop_netcdf_releases_v4_9_2_release_netcdf_c_docs_nczarr.html

hot007 commented 1 month ago

Cross ref this page https://acdguide.github.io/BigData/data/data-zarr.html but also I think that page should be updated seeing it was written for 4.8.0!

Quick note for this book: Pros - improves read efficiency making analysis more tractable, highly performant with python/dask Cons - doesn't inherently support metadata standards (unless used as a back-end for netCDF), uses A LOT of inodes which can cause quota problems unless you work with zipstore, basically only works with python which means it doesn't meet the I part of FAIR.