search

NCPP / ocgis

OpenClimateGIS is a set of geoprocessing and calculation tools for CF-compliant climate datasets.
Other
70 stars 19 forks source link

Output in NetCDF5 format? #516

Closed ekluzek closed 4 years ago

ekluzek commented 4 years ago

It looks like all NetCDF output is in NetCDF4 format. I'm wondering if outputting in NetCDF5 would be a possibility? How hard would it be to do that? In CESM we have some machines that don't work well with NetCDF4, so we always convert to NetCDF5. We can easily convert format by using "nccopy -k 5 " so it isn't too much of a problem. But, if outputting in NetCDF5 would be easy to do, we would prefer that.

bekozi commented 4 years ago

Did you intend to ask about NetCDF 3? If so, then yes, it is possible. There is no switch on the CLI for NetCDF version, but that can be added easily.

ekluzek commented 4 years ago

In CESM we use either NetCDF 3 or NetCDF-5 and avoid NetCDF4. So we could use a CLI to output NetCDF-3. Since, NetCDF-5 is the latest, is it possible to also support it -- or would that be a bigger project?

bekozi commented 4 years ago

Thanks for the clarification. The equivalent flag for cdf-5 in netcdf4-python is NC_FORMAT_64BIT_DATA. I will add the netcdf4-python format flag as an option to the CLI which will allow version 5 (or 3) support.

bekozi commented 4 years ago

@ekluzek I should have asked - should this format be the default?

ekluzek commented 4 years ago

Well for CESM it would be helpful to have NetCDF-5 as the default. But, the ability to change it easily is the most critical thing.

The question for what the default should be though, depends more on the entire community using ocgis. User's are often jolted when the default behavior changes. If you change the default that will help CESM, but might hinder someone else. Are there other large groups of users you could easily query? You could add the ability to set the format as the first step, and then change the default later. Maybe even add a warning that we plan to change the default format to NetCDF-5 in a future version?

bekozi commented 4 years ago

All good points. The plan was to leave the default format as the netCDF4-python default in ocgis generally (this can be changed using environment variables or at write time). I don't think it makes sense to switch defaults at that low of level.

Switching the default for the chunked regridding, however, seems reasonable since NETCDF3_64BIT_OFFSET is the default weight file format in ESMF. What do you think?

ekluzek commented 4 years ago

Makes sense to me then.

bekozi commented 4 years ago

I'll make the switch!

bekozi commented 4 years ago

@ekluzek The default format has been changed and is available on the master branch.