ec-jrc / lisflood-utilities

LISFLOOD OS - Utilities
https://ec-jrc.github.io/lisflood
European Union Public License 1.2
22 stars 9 forks source link

Data type and compression in the output of `cutmaps` #42

Open casadoj opened 10 months ago

casadoj commented 10 months ago

The output NetCDF files from cutmaps do not inherit the data type and compression of the original NetCDF files. The results is that the output file size might be excessively large compared with the original file.

For instance, I've used the cutmaps on the GloFAS4 reservoir fill maps (rsfil.nc):

cutmaps -f ./path_in/ -c "-114.125 -103.475 41.925 50.875" -o ./path_out/

The original files are the daily global maps for every year in the GloFAS long run; their size is approximately 300 Mb. The resulting maps represent a small portion of the Mississippi river, however their size is over 100 Mb. I've done a dummy test with Xarray and the resulting maps are 330 kb, instead.

# load dataset
da = xr.open_dataset(path_in / f'rsfil_2019.nc')
# cut the data
da = da.sel(lon=slice(-114.125, -103.475), lat=slice(50.875, 41.925))
# export NetCDF
encoding = {'rsfil': {'dtype': 'float64', 'zlib': True, 'complevel': 4}}
da.to_netcdf(path / f'giuseppe/test/test_{year}.nc', encoding=encoding)