Can't install/download fesom2 with current develop branch (fine on release)

[koldunovn]

Describe the bug Can't install/download fesom2 with current develop branch. It's working on the release branch: Command:

esm_master install-fesom-2.0

Error:

Traceback (most recent call last):
  File "/home/ollie/nkolduno/.local/bin/esm_master", line 8, in <module>
    sys.exit(main())
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/esm_master/cli.py", line 76, in main
    main_flow(parsed_args, target)
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/esm_master/esm_master.py", line 36, in main_flow
    complete_setup = SimulationSetup(user_config=user_config)
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/esm_runscripts/sim_objects.py", line 42, in __init__
    self.config = prepare.run_job(self.config)
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/esm_runscripts/prepare.py", line 5, in run_job
    helpers.evaluate(config, "prepare", "prepare_recipe")
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/esm_runscripts/helpers.py", line 68, in evaluate
    config = esm_plugin_manager.work_through_recipe(
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/esm_plugin_manager/esm_plugin_manager.py", line 129, in work_through_recipe
    submodule = getattr(thismodule, plugins[workitem]["submodule"])
AttributeError: module 'esm_runscripts' has no attribute 'last_minute'

Same command in the release branch works fine. Maybe it's a bad idea to switch between branches after the installation, but I have also tried to switch branches and reinstall with the same effect. To Reproduce

• Install esm_tools according to docs
• Execute esm_master install-fesom-2.0

System (please complete the following information):

• Supercomputer: ollie
• Version:

  esm_master
  ├─ version: 5.0.1
  ├─ path: /home/ollie/nkolduno/.local/lib/python3.8/site-packages
  ├─ branch:
  └─ tags:

[koldunovn]

Sorry, here is the complete list with versions:

Versions

``` esm_calendar ├─ version: 5.0.0 ├─ path: /home/ollie/nkolduno/.local/lib/python3.8/site-packages ├─ branch: └─ tags: esm_database ├─ version: 5.0.0 ├─ path: /home/ollie/nkolduno/.local/lib/python3.8/site-packages ├─ branch: └─ tags: esm_environment ├─ version: 5.0.0 ├─ path: /home/ollie/nkolduno/.local/lib/python3.8/site-packages ├─ branch: └─ tags: esm_master ├─ version: 5.0.1 ├─ path: /home/ollie/nkolduno/.local/lib/python3.8/site-packages ├─ branch: └─ tags: esm_motd ├─ version: 5.0.2 ├─ path: /home/ollie/nkolduno/.local/lib/python3.8/site-packages ├─ branch: └─ tags: esm_parser ├─ version: 5.0.4 ├─ path: /home/ollie/nkolduno/.local/lib/python3.8/site-packages ├─ branch: └─ tags: esm_plugin_manager ├─ version: 5.0.0 ├─ path: /home/ollie/nkolduno/.local/lib/python3.8/site-packages ├─ branch: └─ tags: esm_profile ├─ version: 5.0.0 ├─ path: /home/ollie/nkolduno/.local/lib/python3.8/site-packages ├─ branch: └─ tags: esm_rcfile ├─ version: 5.0.0 ├─ path: /home/ollie/nkolduno/.local/lib/python3.8/site-packages ├─ branch: └─ tags: esm_runscripts ├─ version: 5.0.14 ├─ path: /home/ollie/nkolduno/.local/lib/python3.8/site-packages ├─ branch: └─ tags: esm_tools ├─ version: 5.0.13 ├─ path: /home/ollie/nkolduno/ESM/esm_tools ├─ branch: develop └─ tags: v5.0.13-28-gdb98fcd-dirty esm_version_checker ├─ version: 5.1.2 ├─ path: /home/ollie/nkolduno/.local/lib/python3.8/site-packages ├─ branch: └─ tags: ```

[mandresm]

Currently, if you use develop branch in esm_tools you'll also need develop branch in the following packages:

• esm_runscripts
• esm_parser

In order to get develop in these packages you need to perform an editable installation of those:

1. cd <your_desired_location> (avoid to navigate and clone into the esm_tools package)
2. git clone https://github.com/esm-tools/<the_package_of_choice>
3. cd <the_package_of_choice>
4. pip install -e .

One question I have, are you trying out ESM-Tools (for which you could simply use release branch for all the packages) or are you planning on starting a contribution?

[mandresm]

Also an additional note, seems that esm_runscripts for the develop branch is currently broken (I found out 15 min ago) and I am working on fixing it. Once it is fixed, you'll need to pull from the packages installed in editable mode.

[koldunovn]

Thanks a lot, @mandresm ! It make sense that one should use most recent versions of all of packages to use the latest features.

I was thinking of adding fesom2.1 to esm_tools, but not sure yet if I would have to do some development of it, or just editing a couple of configs will do the job :)

[mandresm]

@koldunovn , was the issue solved?

[denizural]

Hi @koldunovn, as the solution is both merged to the release and develop branches esm_versions upgrade should take you to the latest version.

[koldunovn]

Thanks a lot! I will check today.

[koldunovn]

I actually get the new error:

/home/ollie/nkolduno/ESM/esm_tools/esm_parser/esm_parser/esm_parser.py:550: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if k is not "debug_info":
/home/ollie/nkolduno/ESM/esm_tools/esm_parser/esm_parser/esm_parser.py:1487: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if level is "keys" and isinstance(right, dict):
Traceback (most recent call last):
  File "/home/ollie/nkolduno/.local/bin/esm_master", line 8, in <module>
    sys.exit(main())
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/esm_master/cli.py", line 15, in main
    check_all_esm_packages()
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/esm_motd/esm_motd.py", line 99, in check_all_esm_packages
    installed_packages_and_versions = get_version_numbers()
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/esm_motd/esm_motd.py", line 117, in get_version_numbers
    tool_mod = importlib.import_module(tool)
  File "/home/ollie/nkolduno/miniconda3/envs/esm/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/esm_environment/__init__.py", line 7, in <module>
    from .esm_environment import *
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/esm_environment/esm_environment.py", line 10, in <module>
    import esm_parser
  File "/home/ollie/nkolduno/ESM/esm_tools/esm_parser/esm_parser/__init__.py", line 9, in <module>
    from .esm_parser import *
  File "/home/ollie/nkolduno/ESM/esm_tools/esm_parser/esm_parser/esm_parser.py", line 113, in <module>
    FUNCTION_PATH = esm_rcfile.EsmToolsDir("FUNCTION_PATH")
AttributeError: module 'esm_rcfile' has no attribute 'EsmToolsDir'

It might well be that I screwed up my installation. Do you have an instruction on how to install everything from scratch in development mode? :)

[mandresm]

There has been some updates into the development branches. Pulling from those packages installed manually (esm_runscripts, esm_parser, esm_tools, esm_rcfile) should be enough. Let me know if that does not work.

[koldunovn]

I guess the question for me is how to properly install the packages manually. What I did is to remove everything with

esm_versions clean

switch to develop branch of esm_tools and run installation with ./install.sh. I guess this pools all other packages and install them in my case to /home/ollie/nkolduno/.local/lib/python3.8/site-packages/.

Then I clone each of the packages you suggested (esm_runscripts, esm_parser, esm_rcfile) change to develop branch for each of them and install with pip install -e .. This installs them into my /home/ollie/nkolduno/miniconda3/envs/esm/lib/python3.8/site-packages/ folder, which is the folder of my esm conda environment. The error persists, and from the trace back I see that for packages I have installed manually the versions from conda directory are used.

I guess the esm_tools should be installed first, since it pools release branches without asking and re-installs (override) the packages I try to install manually, at least this is my impression...

Traceback (most recent call last):
  File "/home/ollie/nkolduno/.local/bin/esm_master", line 8, in <module>
    sys.exit(main())
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/esm_master/cli.py", line 76, in main
    main_flow(parsed_args, target)
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/esm_master/esm_master.py", line 36, in main_flow
    complete_setup = SimulationSetup(user_config=user_config)
  File "/home/ollie/nkolduno/miniconda3/envs/esm/lib/python3.8/site-packages/esm_runscripts/sim_objects.py", line 42, in __init__
    self.config = prepare.run_job(self.config)
  File "/home/ollie/nkolduno/miniconda3/envs/esm/lib/python3.8/site-packages/esm_runscripts/prepare.py", line 5, in run_job
    helpers.evaluate(config, "prepare", "prepare_recipe")
  File "/home/ollie/nkolduno/miniconda3/envs/esm/lib/python3.8/site-packages/esm_runscripts/helpers.py", line 68, in evaluate
    config = esm_plugin_manager.work_through_recipe(
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/esm_plugin_manager/esm_plugin_manager.py", line 129, in work_through_recipe
    submodule = getattr(thismodule, plugins[workitem]["submodule"])
AttributeError: module 'esm_runscripts' has no attribute 'last_minute'

[mandresm]

It all sounds correct. Could you paste here the output of esm_versions check?

[koldunovn]

Well it gives me another error (it worked before):

Traceback (most recent call last):
  File "/home/ollie/nkolduno/.local/bin/esm_versions", line 8, in <module>
    sys.exit(main())
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/esm_version_checker/cli.py", line 262, in check
    attr_dict = get_esm_package_attributes(tool)
  File "/home/ollie/nkolduno/.local/lib/python3.8/site-packages/esm_version_checker/cli.py", line 175, in get_esm_package_attributes
    v2 = config['bumpversion']['current_version']
  File "/home/ollie/nkolduno/miniconda3/envs/esm/lib/python3.8/configparser.py", line 960, in __getitem__
    raise KeyError(key)
KeyError: 'bumpversion'

[mandresm]

I think this was finally solved over a Webex meeting, so I'll close the issue now.

[koldunovn]

Thanks a lot Miguel! The solution was to:

• Remove everything that was installed before (in conda environment and in .local)
• clone esm_tools, change to develop branch and install it with install.sh
• Then clone, change to develop and install with pip install -e . the following packages in THIS SPECIFIC ORDER: esm_runscripts, esm_parser, esm_rcfile.
• Check with esm_versions check each time, that the package installed locally, in the directory where you clone it (not in .local or conda environment)
• ....
• profit :)