ecmwf / cfgrib

A Python interface to map GRIB files to the NetCDF Common Data Model following the CF Convention using ecCodes
Apache License 2.0
407 stars 77 forks source link

have cfgrib exit on skipping variable #345

Closed raybellwaves closed 1 year ago

raybellwaves commented 1 year ago

What happened?

Reading in ECMWF data on s3. Seeing

skipping variable: paramId==167 shortName='t2m'
Traceback (most recent call last):
  File "/Users/ray/miniforge3/envs/wxtools/lib/python3.10/site-packages/cfgrib/dataset.py", line 680, in build_dataset_components
    dict_merge(variables, coord_vars)
  File "/Users/ray/miniforge3/envs/wxtools/lib/python3.10/site-packages/cfgrib/dataset.py", line 611, in dict_merge
    raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='heightAboveGround' value=Variable(dimensions=(), data=10.0) new_value=Variable(dimensions=(), data=2.0)

In my terminal but the code continues. I would like it to exit.

Not sure why it says Traceback is behaves more like a warning than as error

What are the steps to reproduce the bug?


import fsspec
import xarray as xr

file = fsspec.open_local("simplecache::s3://ecmwf-forecasts/20230727/00z/0p4-beta/oper/20230727000000-0h-oper-fc.grib2", s3={"anon": True}, filecache={"cache_storage": "/tmp/files"})
ds = xr.open_dataset(file, engine="cfgrib")

skipping variable: paramId==167 shortName='t2m'
Traceback (most recent call last):
  File "/Users/ray/miniforge3/envs/wxtools/lib/python3.10/site-packages/cfgrib/dataset.py", line 680, in build_dataset_components
    dict_merge(variables, coord_vars)
  File "/Users/ray/miniforge3/envs/wxtools/lib/python3.10/site-packages/cfgrib/dataset.py", line 611, in dict_merge
    raise DatasetBuildError(
cfgrib.dataset.DatasetBuildError: key present and new value is different: key='heightAboveGround' value=Variable(dimensions=(), data=10.0) new_value=Variable(dimensions=(), data=2.0)
>>> 

### Version

0.9.10.4

### Platform (OS and architecture)

mac

### Relevant log output

_No response_

### Accompanying data

_No response_

### Organisation

_No response_
raybellwaves commented 1 year ago

I see the error is displayed via

https://github.com/ecmwf/cfgrib/blob/master/cfgrib/dataset.py#L656

Need to see how to pass that to xarray

Think i got it

import cfgrib
import sys
import fsspec
import xarray as xr

file = fsspec.open_local("simplecache::s3://ecmwf-forecasts/20230727/00z/0p4-beta/oper/20230727000000-0h-oper-fc.grib2", s3={"anon": True}, filecache={"cache_storage": "/tmp/files"})
ds = xr.open_dataset(file, engine="cfgrib")

try:
    ds = xr.open_dataset(file, engine="cfgrib", backend_kwargs={"errors": "raise"})
except cfgrib.dataset.DatasetBuildError as e:
    print(f"Error occurred: {e}")
    sys.exit(1)
iainrussell commented 1 year ago

Thanks Ray, we're working on writing some proper documentation written for cfgrib - this is a good example of a hidden feature!