search

ESMValGroup / ESMValCore

ESMValCore: A community tool for pre-processing data from Earth system models in CMIP and running analysis scripts.
https://www.esmvaltool.org
Apache License 2.0
42 stars 38 forks source link

Configuration file merge fails when `Path` objects are used as dictionary keys #2577

Closed schlunma closed 2 weeks ago

schlunma commented 2 weeks ago

Using multiple configuration file with new syntax for rootpath

rootpath:
  CMIP6:
    /path/to/data: ESGF

fails in dask.config.merge:

ERROR:esmvalcore._main:Program terminated abnormally, see stack trace below for more information:
Traceback (most recent call last):
  File "/home/manuel/ESMValCore/esmvalcore/_main.py", line 618, in run
    fire.Fire(ESMValTool())
  File "/home/manuel/micromamba/envs/esm/lib/python3.12/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/manuel/micromamba/envs/esm/lib/python3.12/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/home/manuel/micromamba/envs/esm/lib/python3.12/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/manuel/ESMValCore/esmvalcore/_main.py", line 427, in run
    CFG.update_from_dirs([cli_config_dir])
  File "/home/manuel/ESMValCore/esmvalcore/config/_config_object.py", line 467, in update_from_dirs
    merged_config_dict = dask.config.merge(self, new_config_dict)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/manuel/micromamba/envs/esm/lib/python3.12/site-packages/dask/config.py", line 172, in merge
    update(result, d)
  File "/home/manuel/micromamba/envs/esm/lib/python3.12/site-packages/dask/config.py", line 132, in update
    update(
  File "/home/manuel/micromamba/envs/esm/lib/python3.12/site-packages/dask/config.py", line 132, in update
    update(
  File "/home/manuel/micromamba/envs/esm/lib/python3.12/site-packages/dask/config.py", line 127, in update
    k = canonical_name(k, old)
        ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/manuel/micromamba/envs/esm/lib/python3.12/site-packages/dask/config.py", line 75, in canonical_name
    altk = k.replace("_", "-") if "_" in k else k.replace("-", "_")
                                  ^^^^^^^^
TypeError: argument of type 'PosixPath' is not iterable

The reason is that dask.config.merge cannot handle Path object as dictionary keys.

I will open a PR with a fix.

valeriupredoi commented 2 weeks ago

this is prob worth pinging Dask for, actually, it looks to me they are missing a basic check on what object they are parsing - bit of a rookie mistake on their part :grin: