search

ecmwf / cfgrib

A Python interface to map GRIB files to the NetCDF Common Data Model following the CF Convention using ecCodes
Apache License 2.0
407 stars 77 forks source link

Open from s3 #297

Open peterdudfield opened 2 years ago

peterdudfield commented 2 years ago

I would like to open a grib file directly from s3.

fsspec is quite handy at opening files from a variety of different file systems

I'm happy to have a go at this myself, just wanted to flag this incase anyone has done this, or its already done

peterdudfield commented 2 years ago

I see it has come up here - https://github.com/ecmwf/cfgrib/issues/198

theoCREALP commented 2 years ago

I'm actually trying to do the same. I'd like to extract a variable from a (large) grib file on a s3 bucket without downloading the file.

Did you find a solution?

raybellwaves commented 2 years ago

Can this be closed?

The linked issue seems to solve this.

There is also https://stackoverflow.com/questions/66229140/xarray-read-remote-grib-file-on-s3-using-cfgrib

theoCREALP commented 2 years ago

It seems to me the link provided above does download the file locally in a temporary folder before accessing it. So I'd say it isn't a solution to this problem.

A potential solution may be using datacube : https://datacube-core.readthedocs.io/en/stable/ops/indexing.html#indexing-data-on-amazon-aws-s3