search

ecmwf / cfgrib

A Python interface to map GRIB files to the NetCDF Common Data Model following the CF Convention using ecCodes
Apache License 2.0
407 stars 77 forks source link

Unable to read index file with xarray #292

Closed mchantry closed 2 years ago

mchantry commented 2 years ago

Unix machine, cfgrib 0.9.10.0 xarray 2022.3.0

Since updating to the latest versions of cfgrib & xarray when I open any grib for the second time [xr.open_dataset(fname,engine='cfgrib') ] I get

Ignoring index file 'sfc_unstructured_2018_12.grib.923a8.idx' incompatible with GRIB file I have ensured that the index file has been made with the same version of cfgrib/xarray. This error cannot be duplicated with, for example, xarray=0.16.2, cfgrib=0.9.9.1.dev0.

As an example, I can duplicate this behaviour with this grib file https://github.com/ecmwf/climetlab/raw/main/docs/examples/test.grib

iainrussell commented 2 years ago

Hi @mchantry thanks for the report. Indeed, the index mechanism was broken, but the version in the master branch now works, at least when the path is given as a string - it still will not work with a pathlib obeject, but I will create a separate issue for that because it raises other issues. Is this enough to have the fix in the master branch, or do you need a PyPi release?

mchantry commented 2 years ago

@iainrussell if it's not too much trouble a PyPi release would be super. Thanks for getting a fix so quickly.

mchantry commented 2 years ago

Actually, there's no real rush as I can use the older version for now. I can wait for the next version.

iainrussell commented 2 years ago

Hi @mchantry oops, didn't see your last message until now - too late, I just released it! :) Version 0.9.10.1. There were quite a few fixes in there, so it's good to get it done.

mchantry commented 2 years ago

@iainrussell thanks very much.

YolandaKok commented 1 year ago

Hello @iainrussell . I've been experiencing the same issue using the cfgrib=0.9.10.3 and xarray=2022.12.0 versions. I am getting that output: Ignoring index file 'test.grib.02ccc.idx' incompatible with GRIB file whenever I am trying to reopen the same files using the open_mfdataset function.

mishooax commented 1 year ago

Hi @iainrussell - I'm hitting the same error as Mat when trying to re-open an indexed GRIB file with open_mfdataset. I pass in a list of strings (file paths) as arguments to open_mfdataset.

xarray                    2022.10.0          pyhd8ed1ab_0    conda-forge
cfgrib                    0.9.10.2           pyhd8ed1ab_0    conda-forge

Error message:

Ignoring index file 'era5_o160_1970.grib.923a8.idx' incompatible with GRIB file
jcreyess commented 9 months ago

If somebody is still dealing with this error, I found a solution:

path_to_grib = 'path/to/file' os.chdir(path_to_grib) list_files = glob.glob('*.grib')

ds = xr.open_dataset(list_files[0], engine = 'cfgrib') print('dataset open...') ds.close() print("dataset closed")

Do not pass the grib file location with a path. Instead of it, change your current directory to the actual file location and from that point read your dataset.