Open malmans2 opened 5 months ago
@malmans2, thank you for reporting this issue. I agree that using xarray_open_mfdataset
consistently would be a good idea. This will be fixed in the next release.
Also related to this issue is the following comment from @malmans2 in #375:
just wanted to provide more details about the use we are doing as you mentioned that we should not import the reader class and a new method will be added:
if isinstance(earthkit_ds, GRIBReader):
xr_ds = earthkit_ds.to_xarray(xarray_open_dataset_kwargs={"squeeze": False, "chunks": {}})
elif isinstance(earthkit_ds, CSVReader):
xr_ds = ds.to_xarray(pandas_read_csv_kwargs=...)
elif ...:
...
else:
xr_ds = earthkit_ds.to_xarray(xarray_open_mfdataset_kwargs=...)
What happened?
Some backends use
xr.open_dataset
whereas others usexr.open_mfdataset
.Because of that, our code does not work seamlessly with all datasets. As
xr.open_mfdataset
is more general and implements more functionalities, would it be possible to use it everywhere?There's also another important downside. The behaviour of
xr.open_dataset
andxr.open_mfdataset
is not identical with single files. For example,xr.open_mfdataset
uses dask by default whereasxr.open_dataset
does not (you'd have to explicitly pass the argumentchunks={}
).What are the steps to reproduce the bug?
Version
0.7.0
Platform (OS and architecture)
Linux eqc-quality-tools.eqc.compute.cci1.ecmwf.int 5.14.0-362.8.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Nov 8 17:36:32 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Relevant log output
Accompanying data
No response
Organisation
B-Open / CADS-EQC