Efficiently access NetCDF files in S3 - Githubissues

xcube-dev / xcube

xcube is a Python package for generating and exploiting data cubes powered by xarray, dask, and zarr.

https://xcube.readthedocs.io/

MIT License

201 stars 20 forks source link

Efficiently access NetCDF files in S3 #928

Closed forman closed 8 months ago

forman commented 8 months ago

Is your feature request related to a problem? Please describe.

We need to efficiently access a huge archive of climate data files in NetCDF format stored in a S3-compatible object storage via the data store framework. The data files are indexed using kerchunk.

Describe the solution you'd like

Add a new filesystem data store that supports the reference protocol.

Additional context

https://climate.esa.int/