Unidata / netcdf-java

The Unidata netcdf-java library
https://docs.unidata.ucar.edu/netcdf-java/current/userguide/index.html
BSD 3-Clause "New" or "Revised" License
142 stars 68 forks source link

S3 Grib support #1177

Open tdrwenski opened 1 year ago

tdrwenski commented 1 year ago

Currently, using GRIB files in an ObjectStore don't work correctly due to the index files (.gbx9) being assumed to be local files. No index file is created if the expected one is not found, since it tries to create a directory using the s3 path which fails.

Expected behavior:

The first and last bullet require creating or updating objects in the ObjectStore, which can be handled through MFile. We could also choose to start with only using a local cache for index files.

WeatherGod commented 1 year ago

Another thing to consider is that (in my opinion) an S3 source of grib files will be from places like ECMWF or one of the Open Data buckets, which is read-only for end-users.