ecmwf / cfgrib

A Python interface to map GRIB files to the NetCDF Common Data Model following the CF Convention using ecCodes
Apache License 2.0
400 stars 77 forks source link

Add support for saving the index produced with a full GRIB scan at open. #20

Closed alexamici closed 5 years ago

alexamici commented 6 years ago

At the moment every time a GRIB file is opened cfgrib needs to scan all the messages in the file to build the index that is then used to compute the values of the coordinates and build the hypercube representation of the variables.

Worse, when opening a GRIB file with the convenience function open_datasets the index is discarded every time the recursive call fails and the expensive file scan is done again.

Proposed implementation requirements for the feature are:

iainrussell commented 6 years ago

I think it's worth mentioning that it probably should not fail if it cannot write the index file - the GRIB file could be in a read-only directory, in which case it would not be possible to write the index to the same place.

alexamici commented 5 years ago

The implementation is still wasteful and very fragile, but the index is saved to and read from disk.

ejhyer commented 2 years ago

I just want to bump @iainrussell point above-- at present, it does not fail when reading from a read-only directory, but it produces a message spew that certainly resembles an I/O error. Versions: cfgrib 0.9.9.1 pyhd8ed1ab_1 conda-forge eccodes 2.23.0 h7621a5c_0 conda-forge python-eccodes 2021.03.0 py39hce5d2b2_2 conda-forge xarray 0.18.2 pyhd8ed1ab_0 conda-forge