Support chunking of returned datasets

xcube-dev / xcube-cds

An xcube plugin to generate data cubes from the Climate Data Store (CDS) API

MIT License

3 stars 1 forks source link

Support chunking of returned datasets #42

Open pont-us opened 3 years ago

pont-us commented 3 years ago

Currently, the CDS Store does not specify any chunking when opening downloaded NetCDF files. This means that the entire datasets are always loaded into memory (see open_dataset and open_mfdataset documentation). The store should provide some option to chunk datasets. Mirroring NetCDF chunking would be a good start, but see caveats in this discussion.

forman commented 3 years ago

For implementation refer to https://github.com/bcdev/nc2zarr/blob/main/nc2zarr/opener.py#L91, method _prefetch_chunk_sizes().