search

intake / intake-esm

An intake plugin for parsing an Earth System Model (ESM) catalog and loading assets into xarray datasets.
https://intake-esm.readthedocs.io
Apache License 2.0
138 stars 47 forks source link

Support for a catalog with mixed netcdf and zarr and other formatted assets #615

Open wachsylon opened 1 year ago

wachsylon commented 1 year ago

Description

It was a nice feature of intake-esm to allow catalogs with combined sources, e.g. data from local disk AND zarr data from cloud by providing a column for the format and let intake decide which xarray backend kwargs are required. This seems to be not possible anymore without BACKEND specific keyword arguments (cdf_kwargs, zarr_kwargs) in the to_dataset_dict function.

What I Did

I used to call the to_dataset_dict function such that I provide both keys, e.g.:

to_dataset_dict(cdf_kwargs=dict(chunks=dict(time=1)),
                             zarr_kwargs=dict(consolidated=True)
                             )

This should be deprecated. If there is only xarary_open_kwargs, the consolidated=True is also passed to the netcdf4 backend and causes:

TypeError: NetCDF4BackendEntrypoint.open_dataset() got an unexpected keyword argument 'consolidated'

Version information: output of intake_esm.show_versions()

'2023.4.20'