search

IQuOD / Formats

Data format development
1 stars 1 forks source link

Which NetCDF version and options to use? #5

Open mhidas opened 6 years ago

mhidas commented 6 years ago

Should IQuOD products be stored in NetCDF3 or NetCDF4 files? If the latter, should we use optional features such as chunking and compression to reduce the file sizes?

Compression would be essential for mutli-profile files with a simple multi-dimensional structure, leading to many fill values, and would also significantly reduce the size of ragged-array files (see https://github.com/IQuOD/Formats/issues/4#issuecomment-380132590).

For a single-profile format, we may as well use NetCDF3 for maximum complatibility.

mhidas commented 6 years ago

Tim Boyer suggests using netCDF4 because it has been around long enough that most people have adopted it, and has useful features that netCDF3 doesn't (compression, ability to store codes instead of code values to avoid repeating string values such as references).

BecCowley commented 6 years ago

I'm taking part in the GTSPP meeting - Chris Paver and Joaquin (both at NCEI) are talking about the development of NETCDF templates at NCEI. Tim Boyer would obviously be aware of this effort. Can we follow up with Chris to review the templates to see how useful they are (he is seeking feedback). Joaquin Trinanes has started with these templates for Gtspp (primarily XBT, CTD, drifters data) and developed on this.