NCAS-CMS / cf-python

A CF-compliant Earth Science data analysis library
http://ncas-cms.github.io/cf-python
MIT License
126 stars 19 forks source link

Reduce CFA file size with small CFA variable HDF chunks #739

Closed davidhassell closed 8 months ago

davidhassell commented 8 months ago

CFA netCDF variables can be very small, much smaller than a default HDF chunk size (= 1 MiB). when there are a lot of these, this can unnecessarily inflate the output CFA netCDF file size.

We could reset the HDF chunk size for these variables to be the variable shape. Even a very large aggregation has a relatively small CFA variables (by design!), so this shouldn't ever be a problem.