search

BSC-ES / autosubmit-config-parser

Library used to read Autosubmit 4 experiment data.
3 stars 1 forks source link

Not all valid YAML files are supported by Autosubmit at the moment #15

Open LuiggiTenorioK opened 1 year ago

LuiggiTenorioK commented 1 year ago

In GitLab by @kinow on May 31, 2023, 12:50

( This is a template for bug reports. Please fill in the relevant information and delete the rest. )

Hello @dbeltrankyl and @kinow,

Autosubmit Version

Latest & latest autosubmit4-config-parser

Expid affected(If applicable)

NA

Which task has issues? Where is the log(If applicable)

NA

Summary

See linked ticket in the parser: https://earth.bsc.es/gitlab/ces/autosubmit4-config-parser/-/issues/3

Not all valid YAML files are supported by Autosubmit. This could be an issue, for example, if users are assuming they can use existing YAML configuration files, and if those files use something like an array with dictionaries.

Found while working on the RO-Crate issue this morning autosubmit#929

Steps to reproduce

Use anything like this in an Autosubmit configuration:

ROCRATE:
  INPUT:
    - name: "DATELIST"
      value: "%EXPERIMENT.DATELIST%"
      additionalType: "String"
      valueRequired: "True"

What is the current bug behavior?

(autosubmit4) kinow@ranma:~/Development/python/workspace/autosubmit4-config-parser$ autosubmit archive --rocrate a008 && unzip -p ~/autosubmit/a008/rocrate.zip ro-crate-metadata.json > ~/autosubmit/a008/ro-crate-metadata.json && subl ~/autosubmit/a008/ro-crate-metadata.json
Autosubmit is running with 4.0.0b
[ERROR] Trace: unhashable type: 'dict'
Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/autosubmit/bin/autosubmit", line 36, in main
    Autosubmit.parse_args()
  File "/home/kinow/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 713, in parse_args
    return Autosubmit.rocrate(args.expid)
  File "/home/kinow/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 4231, in rocrate
    as_configuration.reload(True)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 1382, in reload
    custom_conf_pre = self.load_custom_config_section({}, filenames_to_load["PRE"])
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 1338, in load_custom_config_section
    current_data_pre,current_data_post = self.load_custom_config(current_data, filenames_to_load)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 1305, in load_custom_config
    current_data_pre,current_data_post = self.load_config_folder(copy.deepcopy(current_data),filename)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 612, in load_config_folder
    return self.load_custom_config(current_data, filenames_to_load)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 1317, in load_custom_config
    current_data_pre = self.unify_conf(current_data_pre,self.load_custom_config_section(copy.deepcopy(current_data), filenames_to_load_level["PRE"]))
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 1338, in load_custom_config_section
    current_data_pre,current_data_post = self.load_custom_config(current_data, filenames_to_load)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 1305, in load_custom_config
    current_data_pre,current_data_post = self.load_config_folder(copy.deepcopy(current_data),filename)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 612, in load_config_folder
    return self.load_custom_config(current_data, filenames_to_load)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 1310, in load_custom_config
    current_data = self.substitute_dynamic_variables(self.unify_conf(self.substitute_dynamic_variables(current_data), self.load_config_file(current_data,filename)))
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 583, in load_config_file
    return self.unify_conf(current_folder_data,self.deep_normalize(new_file.data))
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 657, in unify_conf
    current_data = self.deep_update(current_data, new_data)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 490, in deep_update
    tmp = self.deep_update(unified_config.get(key, {}), val)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 494, in deep_update
    if current_list != set(val):
TypeError: unhashable type: 'dict'
 [CRITICAL] Noneunhashable type: 'dict'
Unhandled error: If you see this message, please report it in Autosubmit's GitLab project

The linked issue in autosubmit4-config-parser has more information.

What is the expected correct behavior?

Either support every valid YAML file, or document & warn users about what is supported or not.

Relevant logs and/or screenshots(if applicable)

Done above, and also in the linked issue in autosubmit4-config-parser.

Any other relevant information(if applicable)

()

LuiggiTenorioK commented 1 year ago

In GitLab by @kinow on May 31, 2023, 12:50

To test some YAML, one can use https://www.yamllint.com/, e.g.

image

LuiggiTenorioK commented 1 year ago

In GitLab by @kinow on May 31, 2023, 12:55

Note too that in CWL it's quite common to have this kind of dict/objects in lists, e.g. https://www.commonwl.org/user_guide/topics/expressions.html

image

LuiggiTenorioK commented 1 year ago

In GitLab by @kinow on May 31, 2023, 12:58

Here's another example.

I went to the ScriptEngine website, and copied their Hello Planet example:

- do:
  - base.context:
      planet: Earth
  - base.echo:
      msg: "Hello, {{planet}}!"

Validated in YAML Lint

image

And saved that file to my conf/scriptengine.yml. Then removed the previous invalid block (the one that I used for this issue, so that it would not raise an issue), and re-executed my command. That ScriptEngine YAML also causes the workflow to fail to create/run/etc if it's found in the conf folder:

image

ScriptEngine is used in EC-Earth 4, I think. So we should probably support their YAML syntax, or let users know up ahead if they will encounter any issues (and maybe give them some workaround instructions too).

LuiggiTenorioK commented 1 year ago

In GitLab by @dbeltrankyl on Jun 1, 2023, 08:56

Yes, right now, not all yaml files are available due to the post-processing that Autosubmit does.

It is a bit tricky to support all formats due to the placeholder substitution mechanism and the fact that there Can be duplicated variables ( in different files ) while yaml standard doesn't allow it

related to autosubmit#1000

LuiggiTenorioK commented 1 year ago

In GitLab by @kinow on Jun 5, 2023, 23:50

mentioned in merge request autosubmit!317

LuiggiTenorioK commented 1 year ago

In GitLab by @kinow on Jul 4, 2023, 13:26

mentioned in issue python-amip-reader#5

LuiggiTenorioK commented 1 year ago

In GitLab by @dbeltrankyl on Sep 21, 2023, 11:51

Hello,

There were some fixes for other issues regarding this in the last weeks, maybe it is worth checking if it is working for the rocrate issue

LuiggiTenorioK commented 1 year ago

In GitLab by @kinow on Sep 21, 2023, 14:21

Thanks for the heads up Dani! I'm almost on my way to the airport to return to Barcelona. My inbox is more or less under control, so I should be able to start working on issues again next week. Will try to have a look at this one, and check if the ScriptEngine YAML files can be parsed by AS, as well as the RO-Crate YAML files & that merge request.

LuiggiTenorioK commented 1 year ago

In GitLab by @kinow on Sep 27, 2023, 10:22

Hi @dbeltrankyl ,

I searched the ScriptEngine documentation, and copied the very first YAML I found, here https://scriptengine.readthedocs.io/en/latest/concepts.html

- base.context:
    planet: Earth
- base.echo:
    msg: "Hello, {{planet}}!"

That file can be parsed by online YAML tools, e.g. https://yaml-online-parser.appspot.com/?url=http%3A%2F%2F

image

But if I create a dummy experiment, using the latest Autosubmit code from master, and copy this YAML file somewhere like conf/se.yml, then autosubmit create fails with:

(autosubmit4) kinow@ranma:~/autosubmit$ autosubmit create a006
/home/kinow/mambaforge/envs/autosubmit4/bin/autosubmit:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').require('autosubmit==4.0.93')
[ERROR] Trace: 'list' object has no attribute 'get'
 [CRITICAL] Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/autosubmit/bin/autosubmit", line 37, in main
    Autosubmit.parse_args()
  File "/home/kinow/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 655, in parse_args
    Autosubmit._init_logs(args, args.logconsole, args.logfile, expid)
  File "/home/kinow/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 797, in _init_logs
    as_conf.reload(force_load=True)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 1387, in reload
    starter_conf = self.unify_conf(starter_conf, self.load_config_file(starter_conf, Path(filename)))
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 586, in load_config_file
    if new_file.data.get("DEFAULT", 7000).get("CUSTOM_CONFIG", None) is not None:
AttributeError: 'list' object has no attribute 'get'
'list' object has no attribute 'get'
Unhandled error: If you see this message, please report it in Autosubmit's GitLab project

I checked out the latest main branch from autosubmit4-config-parser, and re-installed it on my venv with pip install -e .. Tried again, and got

(autosubmit4) kinow@ranma:~/Development/python/workspace/autosubmit4-config-parser$ autosubmit create a006
/home/kinow/mambaforge/envs/autosubmit4/bin/autosubmit:4: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  __import__('pkg_resources').require('autosubmit==4.0.93')
[ERROR] Trace: 'list' object has no attribute 'get'
 [CRITICAL] Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/autosubmit/bin/autosubmit", line 37, in main
    Autosubmit.parse_args()
  File "/home/kinow/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 655, in parse_args
    Autosubmit._init_logs(args, args.logconsole, args.logfile, expid)
  File "/home/kinow/Development/python/workspace/autosubmit/autosubmit/autosubmit.py", line 797, in _init_logs
    as_conf.reload(force_load=True)
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 1395, in reload
    starter_conf = self.unify_conf(starter_conf, self.load_config_file(starter_conf, Path(filename)))
  File "/home/kinow/Development/python/workspace/autosubmit4-config-parser/autosubmitconfigparser/config/configcommon.py", line 589, in load_config_file
    if new_file.data.get("DEFAULT", 7000).get("CUSTOM_CONFIG", None) is not None:
AttributeError: 'list' object has no attribute 'get'
'list' object has no attribute 'get'
Unhandled error: If you see this message, please report it in Autosubmit's GitLab project

Same error, I believe.

LuiggiTenorioK commented 8 months ago

In GitLab by @kinow on Feb 1, 2024, 12:51

This issue might be important again, now for the fondos/phenomena project, where they will need a YAML file for their workflow & application. We will have to check if they need/want to use any feature of YAML that is not supported by AS4 config model (the sooner we identify that, the merrier -- cc @ctena & Carmen [couldn't find her user])

LuiggiTenorioK commented 8 months ago

In GitLab by @dbeltrankyl on Feb 27, 2024, 12:53

mentioned in issue digital-twins/de_340/workflow#235

LuiggiTenorioK commented 7 months ago

In GitLab by @kinow on Mar 5, 2024, 11:10

mentioned in issue autosubmit#1240

LuiggiTenorioK commented 7 months ago

In GitLab by @kinow on Mar 12, 2024, 15:45

mentioned in issue otintopr/visualization#9

LuiggiTenorioK commented 2 months ago

In GitLab by @dbeltrankyl on Aug 13, 2024, 10:53

mentioned in merge request autosubmit4-config-parser!9

LuiggiTenorioK commented 2 months ago

In GitLab by @dbeltrankyl on Aug 19, 2024, 13:05

All code is done in the asconfigparser side https://earth.bsc.es/gitlab/es/autosubmit4-config-parser/-/merge_requests/9

LuiggiTenorioK commented 2 months ago

In GitLab by @dbeltrankyl on Aug 19, 2024, 13:07

moved from autosubmit#1045

LuiggiTenorioK commented 2 months ago

In GitLab by @dbeltrankyl on Aug 19, 2024, 13:07

Moved from autosubmit repo to asconfigparser