Closed jodemaey closed 2 years ago
I see. I reproduced the issue, thanks for the detailed report.
The reason is that your grib files (from mutiple sources) are not compatible.
Previous versions of (0.9.9 and below), we were sending each grib separately to cfgrib. But now, for multi source of grib, climetlab is doing the indexing, computing offsets etc. and presenting to cfgrib something that looks like a unique grib file. If you open such a grib file cfgrib fails with this error.
Nevertheless, we do want to support this use case nicely. I will work on it during the next few days.
We would like to make cfgrib ignore the value of the inconsistent key (namely "dataType" which is sometimes 'pf' and sometimes 'cf'). This is not yet implemented in cfgrib. I added it in climetlab and opened an issue to have it implemented in cfgrib.
Would you mind checking that this solves the issue?
1) Use the current develop branch of climetlab (commit ee697ff or later) 2) Change your TrainingDataForecastSurfacePostProcessed.to_xarray() to add the option to ignore the failing keys as follow:
fcs = self.source.to_xarray(
xarray_open_dataset_kwargs={"backend_kwargs": {"ignore_keys": ["dataType"]}},
**kwargs,
)
It seems to work now indeed:
Does that mean that we need to overload the to_xarray
routine everywhere this might happen?
Yes, the current default behaviour of cfgrib is to raise an error when the GRIB are not consistent. The rationale is that we really do not want to change the grib file on the fly without the user deciding it explicitly. So, yes, the option "ignore_keys" needs to be added everywhere you need it.
Ok thanks, I think that this issue can be closed when there is a new release with the fix then.
version 0.10.3 released.
After implementing this modification, everything seems fine and I will release a new version of the plugin. Thanks for your help.
Hi there,
Since the release 0.10.1, the EUMETNET benchmark plugin is partly broken because apparently cfgrib does not accept anymore to merge the control and perturbed ECMWF forecasts together.
I use the multi source to do that (see here), but when calling
to_xarray
, I get the following issue:So the problem seems to be that it has become more strict about the multiple values for keys, using this function
enforce_unique_attributes
.cfgrib version is
0.9.10
.How can I resolve this issue? Can I tell him to be more forgiving about the attributes uniqueness?
Thank you in advance.
To reproduce, simply install the plugin
pip install climetlab climetlab-eumetnet-postprocessing-benchmark
and run the notebook https://github.com/Climdyn/climetlab-eumetnet-postprocessing-benchmark/tree/main/notebooks/demo_ensemble_forecasts.ipynb .