search

environmental-forecasting / download-toolbox

A toolbox of downloaders for environmental data
MIT License
2 stars 1 forks source link

An option for data compression of netCDF outputs #7

Open bnubald opened 6 months ago

bnubald commented 6 months ago

Since the data downloaded can be of quite significant sizes, it might be helpful to have a flag to optionally enable netCDF's data compression with different levels.

Reason for mentioning is that for the EDS book, where ftp was blocked (Wanted to avoid external download separate to the library demo), I was able to get 96% compression (174MB to 6.6MB with lowest compression level, and 5.7MB with highest compression level) for 3 months of OSI-SAF Sea Ice Concentration data.

Not sure if this should be here, or under IceNet. Applying this to the processed data did affect the time taken for dataset generation of tfrecords in IceNet quite a bit.

JimCircadian commented 6 months ago

Perfect. This is definitely worth doing, thanks @bnubald - should live in the download-toolbox for sure, as that will be building on anything present in the 0.2 version of icenet and this code pathway won't exist in 0.3 of icenet