ProjectPythia / esgf-cookbook

Cookbook for ESGF-hosted datasets
https://projectpythia.org/esgf-cookbook/
Apache License 2.0
2 stars 9 forks source link

Missed dependencies error for `netcdf4` and `h5netcdf` #7

Open lee1043 opened 11 months ago

lee1043 commented 11 months ago

I have created my conda env using the enviroment.yml following the instructions in the README.md, and ran below code which I am getting following error that complains it cannot find netcdf4, h5netcdf.

I resolved it by manually installing those two libraries. I think a possible solution could be adding netcdf4 and h5netcdf as a part of environment.yml

cat = ESGFCatalog(esgf1_indices="esgf-node.llnl.gov")

cat.search(
    experiment_id="historical",
    source_id="CanESM5",
    frequency="mon",
    variable_id=["gpp", "tas", "pr"],
    variant_label="r1i1p1f1",  # addition from the last search
)

dsd = cat.to_dataset_dict()
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/intro-search_AGU_tutorial.ipynb](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/intro-search_AGU_tutorial.ipynb) Cell 13 line 1
----> [1](vscode-notebook-cell:/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/intro-search_AGU_tutorial.ipynb#X15sZmlsZQ%3D%3D?line=0) dsd = cat.to_dataset_dict()

File [~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/intake_esgf/catalog.py:388](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/intake_esgf/catalog.py:388), in ESGFCatalog.to_dataset_dict(self, minimal_keys, ignore_facets, separator, num_threads, quiet, add_measures, operators)
    [386](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/intake_esgf/catalog.py:386) for key, files in ds.items():
    [387](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/intake_esgf/catalog.py:387)     if len(files) == 1:
--> [388](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/intake_esgf/catalog.py:388)         ds[key] = xr.open_dataset(files[0])
    [389](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/intake_esgf/catalog.py:389)     elif len(files) > 1:
    [390](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/intake_esgf/catalog.py:390)         ds[key] = xr.open_mfdataset(sorted(files))

File [~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/xarray/backends/api.py:553](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/xarray/backends/api.py:553), in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, inline_array, chunked_array_type, from_array_kwargs, backend_kwargs, **kwargs)
    [550](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/xarray/backends/api.py:550)     kwargs.update(backend_kwargs)
    [552](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/xarray/backends/api.py:552) if engine is None:
--> [553](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/xarray/backends/api.py:553)     engine = plugins.guess_engine(filename_or_obj)
    [555](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/xarray/backends/api.py:555) if from_array_kwargs is None:
    [556](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/xarray/backends/api.py:556)     from_array_kwargs = {}

File [~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/xarray/backends/plugins.py:197](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/xarray/backends/plugins.py:197), in guess_engine(store_spec)
    [189](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/xarray/backends/plugins.py:189) else:
    [190](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/xarray/backends/plugins.py:190)     error_msg = (
    [191](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/xarray/backends/plugins.py:191)         "found the following matches with the input file in xarray's IO "
    [192](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/xarray/backends/plugins.py:192)         f"backends: {compatible_engines}. But their dependencies may not be installed, see:\n"
    [193](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/xarray/backends/plugins.py:193)         "https://docs.xarray.dev/en/stable/user-guide/io.html \n"
    [194](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/xarray/backends/plugins.py:194)         "https://docs.xarray.dev/en/stable/getting-started-guide/installing.html"
    [195](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/xarray/backends/plugins.py:195)     )
--> [197](https://file+.vscode-resource.vscode-cdn.net/Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/xarray/backends/plugins.py:197) raise ValueError(error_msg)

ValueError: found the following matches with the input file in xarray's IO backends: ['netcdf4', 'h5netcdf']. But their dependencies may not be installed, see:
https://docs.xarray.dev/en/stable/user-guide/io.html 
https://docs.xarray.dev/en/stable/getting-started-guide/installing.html
nocollier commented 11 months ago

I saw this on a few participants, thanks for the report. I think we just need to manually include netcdf4 in the Python installs.

On Mon, Dec 11, 2023 at 8:51 PM Jiwoo Lee @.***> wrote:

I have created my conda env using the enviroment.yml following the instructions in the README.md https://github.com/esgf2-us/esgf-cookbook/blob/main/README.md, and ran below code which I am getting following error that complains it cannot find netcdf4, h5netcdf.

I resolved it by manually installing those two libraries. I think a possible solution could be adding netcdf4 and h5netcdf as a part of environment.yml https://github.com/esgf2-us/esgf-cookbook/blob/main/environment.yml#L17

cat = ESGFCatalog(esgf1_indices="esgf-node.llnl.gov") cat.search( experiment_id="historical", source_id="CanESM5", frequency="mon", variable_id=["gpp", "tas", "pr"], variant_label="r1i1p1f1", # addition from the last search ) dsd = cat.to_dataset_dict()


ValueError Traceback (most recent call last) /Users/lee1043/Documents/Research/git/esgf-cookbook/notebooks/intro-search_AGU_tutorial.ipynb Cell 13 line 1 ----> 1 dsd = cat.to_dataset_dict()

File ~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/intake_esgf/catalog.py:388, in ESGFCatalog.to_dataset_dict(self, minimal_keys, ignore_facets, separator, num_threads, quiet, add_measures, operators) 386 for key, files in ds.items(): 387 if len(files) == 1: --> 388 ds[key] = xr.open_dataset(files[0]) 389 elif len(files) > 1: 390 ds[key] = xr.open_mfdataset(sorted(files))

File ~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/xarray/backends/api.py:553, in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, inline_array, chunked_array_type, from_array_kwargs, backend_kwargs, **kwargs) 550 kwargs.update(backend_kwargs) 552 if engine is None: --> 553 engine = plugins.guess_engine(filename_or_obj) 555 if from_array_kwargs is None: 556 from_array_kwargs = {}

File ~/mambaforge/envs/esgf-cookbook-dev/lib/python3.10/site-packages/xarray/backends/plugins.py:197, in guess_engine(store_spec) 189 else: 190 error_msg = ( 191 "found the following matches with the input file in xarray's IO " 192 f"backends: {compatible_engines}. But their dependencies may not be installed, see:\n" 193 "https://docs.xarray.dev/en/stable/user-guide/io.html \n" 194 "https://docs.xarray.dev/en/stable/getting-started-guide/installing.html" 195 ) --> 197 raise ValueError(error_msg)

ValueError: found the following matches with the input file in xarray's IO backends: ['netcdf4', 'h5netcdf']. But their dependencies may not be installed, see:https://docs.xarray.dev/en/stable/user-guide/io.html https://docs.xarray.dev/en/stable/getting-started-guide/installing.html

— Reply to this email directly, view it on GitHub https://github.com/esgf2-us/esgf-cookbook/issues/7, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKFCB5KTYWAFFWEJL5CAMLYI7PGLAVCNFSM6AAAAABAQ23LVGVHI2DSMVQWIX3LMV43ASLTON2WKOZSGAZTMOJXG43DAMA . You are receiving this because you are subscribed to this thread.Message ID: @.***>