wish: esm_master get-model on arbitrary machine independent on configs/machines

chrisdane commented 3 years ago

Is your feature request related to a problem? Please describe. I wonder if just downloading a model, i.e. esm_master get-*, could be possible on an arbitrary machine not defined in configs/machines. That would make it easier to, e.g., study model code offline. In the current situation, the error

WARNING:root:The yaml file for this computer (<my_local_computer_name>) could not be determined!
WARNING:root:Continuing with generic settings...
...
KeyError: 'Key computer.name was not defined'

stops the call.

Describe the solution you'd like Make the call esm_master get-* independent of the defined configs/machines.

Cheers, Chris

pgierz commented 3 years ago

We used to have a machine called "generic", which pretended it was ollie...not sure if it is still in the configs or not

pgierz commented 3 years ago

Chris, can you put more of that traceback? what exactly is trying to use computer.name?

chrisdane commented 3 years ago

I think its easiest if you esm-master get-fesom-2.0 on you local computer. This is the full error on my local pc:

WARNING:root:The yaml file for this computer (mylocalpc) could not be determined!
WARNING:root:Continuing with generic settings...
Traceback (most recent call last):
  File "~/.local/lib/python3.9/site-packages/esm_parser/esm_parser.py", line 1722, in recursive_get
    result = config_to_search[this_config]
KeyError: 'name'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/.local/lib/python3.9/site-packages/esm_parser/esm_parser.py", line 1344, in resolve_basic_choose
    choice = recursive_get(config, path_to_key)
  File "~/.local/lib/python3.9/site-packages/esm_parser/esm_parser.py", line 1729, in recursive_get
    return recursive_get(result, my_config_elements)
  File "~/.local/lib/python3.9/site-packages/esm_parser/esm_parser.py", line 1724, in recursive_get
    raise ValueError(
ValueError: ("Exactly None! Couldn't find an answer for:", [])

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/.local/bin/esm_master", line 33, in <module>
    sys.exit(load_entry_point('esm-master==5.1.1', 'console_scripts', 'esm_master')())
  File "~/.local/lib/python3.9/site-packages/esm_master/cli.py", line 111, in main
    main_flow(parsed_args, target)
  File "~/.local/lib/python3.9/site-packages/esm_master/esm_master.py", line 56, in main_flow
    complete_setup = SimulationSetup(user_config=user_config)
  File "~/.local/lib/python3.9/site-packages/esm_runscripts/sim_objects.py", line 47, in __init__
    self.config = prepare.run_job(self.config)
  File "~/.local/lib/python3.9/site-packages/esm_runscripts/prepare.py", line 6, in run_job
    helpers.evaluate(config, "prepare", "prepare_recipe")
  File "~/.local/lib/python3.9/site-packages/esm_runscripts/helpers.py", line 100, in evaluate
    config = esm_plugin_manager.work_through_recipe(
  File "~/.local/lib/python3.9/site-packages/esm_plugin_manager/esm_plugin_manager.py", line 138, in work_through_recipe
    config = getattr(submodule, workitem)(config)
  File "~/.local/lib/python3.9/site-packages/esm_runscripts/prepare.py", line 116, in resolve_some_choose_blocks
    choose_blocks(config, blackdict=config._blackdict)
  File "~/.local/lib/python3.9/site-packages/esm_parser/esm_parser.py", line 2417, in choose_blocks
    resolve_basic_choose(
  File "~/.local/lib/python3.9/site-packages/esm_parser/esm_parser.py", line 1347, in resolve_basic_choose
    raise KeyError("Key %s was not defined" % ".".join(path_to_key))
KeyError: 'Key computer.name was not defined'

denizural commented 3 years ago

It looks like it it getting the current hostname (mylocalpc) and can't find the corresponding yaml file obviously. Maybe we can implement the option to use the generic.yaml when the current hostname is not in the supported machines list.

pgierz commented 3 years ago

I'm cleaning up, sorry if this gets "rudely closed"

Why not just do a git clone if you need a particular model without compiling?

jrberlin commented 3 years ago

Hi I am having exactly the same error, just wanting to do some basic tests, such as getting the code of a given model, even adding a configuration file matching the computer name inside config/machines throws the same error , is there any wiki with information on how to add a new machine properly ?

Thanks !

pgierz commented 3 years ago

Hi, "better errors" is something that has been on our list. Sorry you are running into problems.

Can you push whatever you have so far? One of us can then take a look. If the machine file is good, there might be a bit missing in "known machines" (@Miguel or Deniz am I remembering that wrong?)

jrberlin commented 3 years ago

Hi, I only did the following:

copied an existing machine file (mistral.yaml) and renamed to the workstation name (bscearth384.yaml) where I am trying to run in esm_tools/configs/machines, I also added a entry in all_machines.yaml
I added a new configuration that points to a different git repository, I added such option inside fesom-2.0.yaml: file and then I execute the following line : esm_master get-fesom-2.0-bsc and I get this error (pretty similar to the ones mentioned before);

WARNING:root:The yaml file for this computer (mymachine) could not be determined! WARNING:root:Continuing with generic settings... Traceback (most recent call last): File "/home/.local/lib/python3.7/site-packages/esm_parser/esm_parser.py", line 1764, in recursive_get result = config_to_search[this_config] KeyError: 'name'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/.local/lib/python3.7/site-packages/esm_parser/esm_parser.py", line 1365, in resolve_basic_choose choice = recursive_get(config, path_to_key) File "/home/.local/lib/python3.7/site-packages/esm_parser/esm_parser.py", line 1771, in recursive_get return recursive_get(result, my_config_elements) File "/home/.local/lib/python3.7/site-packages/esm_parser/esm_parser.py", line 1767, in recursive_get "Exactly None! Couldn't find an answer for:", my_config_elements ValueError: ("Exactly None! Couldn't find an answer for:", [])

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/.local/bin/esm_master", line 8, in sys.exit(main()) File "/home/.local/lib/python3.7/site-packages/esm_master/cli.py", line 103, in main main_flow(parsed_args, target) File "/home/.local/lib/python3.7/site-packages/esm_master/esm_master.py", line 53, in main_flow complete_setup = SimulationSetup(user_config=user_config) File "/home/.local/lib/python3.7/site-packages/esm_runscripts/sim_objects.py", line 47, in init self.config = prepare.run_job(self.config) File "/home/.local/lib/python3.7/site-packages/esm_runscripts/prepare.py", line 6, in run_job helpers.evaluate(config, "prepare", "prepare_recipe") File "/home/.local/lib/python3.7/site-packages/esm_runscripts/helpers.py", line 101, in evaluate framework_recipe, framework_plugins, config File "/home/.local/lib/python3.7/site-packages/esm_plugin_manager/esm_plugin_manager.py", line 138, in work_through_recipe config = getattr(submodule, workitem)(config) File "/home/.local/lib/python3.7/site-packages/esm_runscripts/prepare.py", line 111, in resolve_some_choose_blocks choose_blocks(config, blackdict=config._blackdict) File "/home/.local/lib/python3.7/site-packages/esm_parser/esm_parser.py", line 2468, in choose_blocks blackdict[model_with_choose] File "/home/.local/lib/python3.7/site-packages/esm_parser/esm_parser.py", line 1368, in resolve_basic_choose raise KeyError("Key %s was not defined" % ".".join(path_to_key)) KeyError: 'Key computer.name was not defined'

denizural commented 3 years ago

Hi @jrberlin, I am also working on this feature since I would like to adapt esm-tools for small computing systems such as faculty workstations for teaching purposes.

There is going to be a --machine <your_machine.yaml> or

machine: <machine-name>
machine_yaml: <path-to-machine-yaml>

in esm executables.

Regarding your previous problem, do you have a repository or fork where I can have a look at what you did? I am reopening this issue since it is a feature that we need to add at some point.

pgierz commented 3 years ago

Can you post your versions: esm-tools should have installed the version checker for you, just try esm_versions check, and put whatever you have online? Hopefully you have $HOME/.local/bin in your path, and if you do an ls there, you should see the various esm programs.

I don't have access to your specific machine, so I will only be able to do "minimal" tests with a dummy machine running Ubuntu, but it should translate to your bscearth384. I guess you have Centos or Redhad or something, but that won't matter for your particular problem. Make yourself a branch, call it jrberlin, or, pick your favorite name ;-) That'll help me figure out whats going on. computer.name being unknown is hopefully just something small

It might be that you get some sort of strange "you are not allowed to push" error. If that is the case, you will need to fork everything and put it onto your personal account, but I can help you with that if needed. Also just let me know if you need any other help with git, but I am afraid without a closer look at your actual files, it will be difficult to see what is happening

@denizural, who closed the issue? If I did that by accident, sorry.....

pgierz commented 3 years ago

Yes I closed it in my "cleanup frenzy"...sorry for that.

jrberlin commented 3 years ago

Hi @pgierz , the workstation in question have suse Linux

> esm_versions check

+---------------------+-----------+--------------------------------------------------------+----------+--------------------------+
| package_name        | version   | file                                                   | branch   | tags                     |
|---------------------+-----------+--------------------------------------------------------+----------+--------------------------|
| esm_calendar        | 5.0.0     | /home/.local/lib/python3.7/site-packages |          |                          |
| esm_database        | 5.0.0     | /home/.local/lib/python3.7/site-packages |          |                          |
| esm_environment     | 5.1.2     | /home/.local/lib/python3.7/site-packages |          |                          |
| esm_master          | 5.1.6     | /home/.local/lib/python3.7/site-packages |          |                          |
| esm_motd            | 5.0.2     | /home/.local/lib/python3.7/site-packages |          |                          |
| esm_parser          | 5.1.10    | /home/.local/lib/python3.7/site-packages |          |                          |
| esm_plugin_manager  | 5.0.1     | /home/.local/lib/python3.7/site-packages |          |                          |
| esm_profile         | 5.0.0     | /home/.local/lib/python3.7/site-packages |          |                          |
| esm_rcfile          | 5.1.0     | /home/.local/lib/python3.7/site-packages |          |                          |
| esm_runscripts      | 5.1.30    | /home/.local/lib/python3.7/site-packages |          |                          |
| esm_tools           | 5.1.21    | /home/Development/esm_tools              | release  | v5.1.21-1-g31e1200-dirty |
| esm_version_checker | 5.1.5     | /home/.local/lib/python3.7/site-packages |          |                          |
+---------------------+-----------+--------------------------------------------------------+----------+--------------------------+

> ll $HOME/.local/bin
total 64
-rwxr-xr-x 1 jberlin  260 Sep  1 17:38 coloredlogs
-rwxr-xr-x 1 jberlin  261 Sep  1 17:39 esm_database
-rwxr-xr-x 1 jberlin  259 Sep  1 17:39 esm_master
-rwxr-xr-x 1 jberlin  267 Sep  1 17:39 esm_plugins
-rwxr-xr-x 1 jberlin  263 Sep  1 17:39 esm_runscripts
-rwxr-xr-x 1 jberlin  268 Sep  1 17:38 esm_versions
-rwxr-xr-x 1 jberlin  262 Sep  1 17:39 f2py
-rwxr-xr-x 1 jberlin  262 Sep  1 17:39 f2py3
-rwxr-xr-x 1 jberlin  262 Sep  1 17:39 f2py3.7
-rwxr-xr-x 1 jberlin  102 Sep  1 17:39 f90nml
-rwxr-xr-x 1 jberlin  260 Sep  1 17:41 gfw_creator
-rwxr-xr-x 1 jberlin  262 Sep  1 17:38 humanfriendly
-rwxr-xr-x 1 jberlin  266 Sep  1 17:41 nc3tonc4
-rwxr-xr-x 1 jberlin  266 Sep  1 17:41 nc4tonc3
-rwxr-xr-x 1 jberlin  262 Sep  1 17:41 ncinfo
-rwxr-xr-x 1 jberlin  255 Sep  1 17:38 tabulate

our goal is to also do some adaptations to use in other HPCs and also be able to point to different repositories where other models are present

jrberlin commented 3 years ago

Hi,

I think I found one of the problems: the YAML file for the generic machine lacks of the name field, I added it by hand but still having other errors:


/home/model_codes  > esm_master get-fesom-2.0-bsc
WARNING:root:The yaml file for this computer (mymachine) could not be determined!
WARNING:root:Continuing with generic settings...
Traceback (most recent call last):
  File "/home/.local/bin/esm_master", line 8, in <module>
    sys.exit(main())
  File "/home/Earth/jberlin/.local/lib/python3.7/site-packages/esm_master/cli.py", line 103, in main
    main_flow(parsed_args, target)
  File "/home/.local/lib/python3.7/site-packages/esm_master/esm_master.py", line 63, in main_flow
    parsed_args)
  File "/home/.local/lib/python3.7/site-packages/esm_master/task.py", line 91, in __init__
    "compiletime", complete_config, model
  File "/home/.local/lib/python3.7/site-packages/esm_environment/esm_environment.py", line 91, in __init__
    self.add_esm_var()
  File "/home/.local/lib/python3.7/site-packages/esm_environment/esm_environment.py", line 103, in add_esm_var
    self.config["export_vars"]["ENVIRONMENT_SET_BY_ESMTOOLS"] = "TRUE"
TypeError: 'NoneType' object does not support item assignment

I think maybe that dict is not instantiate in that call ?

mandresm commented 2 years ago

After @seb-wahl's fixes to the generic.yaml and rcfile feature deletion, this should be supported now. I tried esm_master get-fesom-2.1 in my laptop and it did the job. Please, reopen the issue if anyone still encounters problems.

esm-tools / esm_tools

wish: esm_master get-model on arbitrary machine independent on configs/machines #344