New minor awicm3 version: stat: path should be string, bytes, os.PathLike or integer, not list

[JanStreffing]

Describe the problem you are facing after implementing a new minor version awicm3 v3.1.1 I got the following error at runtime, after the setup of the work folder is nearly complete:

Traceback (most recent call last):
  File "/home/a/a270092/.local/bin/esm_runscripts", line 33, in <module>
    sys.exit(load_entry_point('esm-tools', 'console_scripts', 'esm_runscripts')())
  File "/home/a/a270092/esm_tools/src/esm_runscripts/cli.py", line 278, in main
    Setup()
  File "/home/a/a270092/esm_tools/src/esm_runscripts/sim_objects.py", line 69, in __call__
    self.prepcompute()
  File "/home/a/a270092/esm_tools/src/esm_runscripts/sim_objects.py", line 161, in prepcompute
    self.config = prepcompute.run_job(self.config)
  File "/home/a/a270092/esm_tools/src/esm_runscripts/prepcompute.py", line 32, in run_job
    config = evaluate(config, "prepcompute", "prepcompute_recipe")
  File "/home/a/a270092/esm_tools/src/esm_runscripts/helpers.py", line 69, in evaluate
    config = esm_plugin_manager.work_through_recipe(
  File "/home/a/a270092/esm_tools/src/esm_plugin_manager/esm_plugin_manager.py", line 141, in work_through_recipe
    config = getattr(submodule, workitem)(config)
  File "/home/a/a270092/esm_tools/src/esm_runscripts/filelists.py", line 1079, in report_missing_files
    config = _check_fesom_missing_files(config)
  File "/home/a/a270092/esm_tools/src/esm_runscripts/filelists.py", line 1116, in _check_fesom_missing_files
    if not os.path.exists(path):
  File "/sw/spack-levante/mambaforge-4.11.0-0-Linux-x86_64-sobz6z/lib/python3.9/genericpath.py", line 19, in exists
    os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not list

To Reproduce checkout branch https://github.com/esm-tools/esm_tools/tree/feat/awicm3.1.1, esm_master install-awicm3-v3.1.1 run a check run on levante using esm_tools/runscripts/awicm3/v3.1.1/awicm3-v3.1.1-levante-TCO95L91-CORE2.yaml

The first commit on branch https://github.com/esm-tools/esm_tools/tree/awicm3-v3.1.1 already causes the issue. So it's not the wild-carding that I'm trying out for the minor release.

[JanStreffing]

I think i have an idea of what happened here. I had alread made a branch with the same name more than half a year ago. I accidentally check that one out, when I was planning to make a new branch. I will try to properly branch off from release and give it another go.

[mandresm]

It's failing in the FESOM specific file checker: _check_fesom_missing_files

It's checking the namelist, that tells me there is something weird in the namelist.config.

[JanStreffing]

Yeah, i went in with the breakpoint:

Run number for internal OpenIFS timekeeping: 1
> /home/a/a270092/esm_tools/src/esm_runscripts/filelists.py(1117)_check_fesom_missing_files()
-> if not os.path.exists(path):
(Pdb) path
'/work/ab0246/a270092/input/fesom2//core2/'
(Pdb) continue
> /home/a/a270092/esm_tools/src/esm_runscripts/filelists.py(1116)_check_fesom_missing_files()
-> breakpoint()
(Pdb) path
'/work/ab0246/a270092/input/fesom2//hydrography/'
(Pdb) continue
> /home/a/a270092/esm_tools/src/esm_runscripts/filelists.py(1117)_check_fesom_missing_files()
-> if not os.path.exists(path):
(Pdb) path
['<', 'tideforcingpath', '>']

I think this variable is missing?

[JanStreffing]

it works when i set:

    add_namelist_changes:
        namelist.config:
            paths:
                tideforcingpath: "${pool_dir}/fesom/tide_forcing/"

in the runscript, but the variable tide_forcing_dir: "${pool_dir}/fesom/tide_forcing/" which should be used by the fesom.2.0 yaml:

namelist_changes:
        namelist.config:
                paths:
                        TideForcingPath: "${tide_forcing_dir}"

Does not seem to work. Instead I get this strange string.

[mandresm]

It seems that for some reason tide_forcing_dir is defined as <tideforcingpath>. That path obviously does not exist. I'm wondering though, why we see this now and not before. What is the value of tide_forcing_dir in the finished config?

[mandresm]

Hey Jan, I'm thinking... are you sure the logic 3.1* works for choose blocks? I've never used that, so it might be that non of your choose are resolve correctly. That could cause the problem you are seeing, as the necessary namelist changes wouldn't be selected (just an hypothesis). If it doesn't work, it is indeed a feature that should be implemented as it will help tremendously for choosing minor versions. In the mean time, you can use the logic I referred to here: https://github.com/esm-tools/esm_tools/pull/991#issuecomment-1623633892

[mandresm]

This would be even a shorter version of the provisional solution I'm proposing: https://github.com/esm-tools/esm_tools/pull/991#issuecomment-1623640351

[JanStreffing]

closed by https://github.com/esm-tools/esm_tools/pull/991