NOAA-EMC / global-workflow

Global Superstructure/Workflow supporting the Global Forecast System (GFS)
https://global-workflow.readthedocs.io/en/latest
GNU Lesser General Public License v3.0
75 stars 168 forks source link

Crash in setting up the global-workflow experiment #2884

Closed jiaruidong2017 closed 2 months ago

jiaruidong2017 commented 2 months ago

What is wrong?

My script can successfully setup the g-w experiment previously. But recently, the script crashed with the error messages below:

...........................................................
Source configs with RUN=gdas
Traceback (most recent call last):
  File "/scratch1/NCEPDEV/global/Jiarui.Dong/JEDI/GlobalWorkflow/global-workflow.snowtest9/workflow/create_experiment.py", line 108, in <module>
    setup_xml.main(setup_xml_args)
  File "/scratch1/NCEPDEV/global/Jiarui.Dong/JEDI/GlobalWorkflow/global-workflow.snowtest9/workflow/setup_xml.py", line 73, in main
    xml = rocoto_xml_factory.create(f'{net}_{mode}', app_config, rocoto_param_dict)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch1/NCEPDEV/global/Jiarui.Dong/JEDI/GlobalWorkflow/global-workflow.snowtest9/workflow/wxflow/factory.py", line 71, in create
    return self._builders[key](*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch1/NCEPDEV/global/Jiarui.Dong/JEDI/GlobalWorkflow/global-workflow.snowtest9/workflow/rocoto/gfs_cycled_xml.py", line 12, in __init__
    super().__init__(app_config, rocoto_config)
  File "/scratch1/NCEPDEV/global/Jiarui.Dong/JEDI/GlobalWorkflow/global-workflow.snowtest9/workflow/rocoto/workflow_xml.py", line 27, in __init__
    task_list = get_wf_tasks(app_config)
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch1/NCEPDEV/global/Jiarui.Dong/JEDI/GlobalWorkflow/global-workflow.snowtest9/workflow/rocoto/workflow_tasks.py", line 19, in get_wf_tasks
    task_obj = tasks_factory.create(app_config.net, app_config, run)  # create Task object based on run
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch1/NCEPDEV/global/Jiarui.Dong/JEDI/GlobalWorkflow/global-workflow.snowtest9/workflow/wxflow/factory.py", line 71, in create
    return self._builders[key](*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch1/NCEPDEV/global/Jiarui.Dong/JEDI/GlobalWorkflow/global-workflow.snowtest9/workflow/rocoto/gfs_tasks.py", line 11, in __init__
    super().__init__(app_config, run)
  File "/scratch1/NCEPDEV/global/Jiarui.Dong/JEDI/GlobalWorkflow/global-workflow.snowtest9/workflow/rocoto/tasks.py", line 53, in __init__
    self.HOMEgfs = self._base['HOMEgfs']
                   ~~~~~~~~~~^^^^^^^^^^^
  File "/scratch1/NCEPDEV/global/Jiarui.Dong/JEDI/GlobalWorkflow/global-workflow.snowtest9/workflow/wxflow/attrdict.py", line 82, in __missing__
    if object.__getattribute__(self, '__frozen'):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'AttrDict' object has no attribute '__frozen'

What should have happened?

The script crashed and setup failed.

What machines are impacted?

Hera

Steps to reproduce

Run the script on hera: /scratch1/NCEPDEV/global/Jiarui.Dong/JEDI/GlobalWorkflow/tools/setup_exp_fail.sh

Additional information

NA

Do you have a proposed solution?

Thanks @RussTreadon-NOAA for his suggestion. As suggested, when I changed from "export HOMEgfs=" to "HOMEgfs=", it successfully built the g-w experiment. I open this issue to alert this behavior.

aerorahul commented 2 months ago

@jiaruidong2017 The setup_xml.py script sources the bash config files and extracts those settings from the underlying user env. If a key such as HOMEgfs exists in the underlying user env., it is removed.

This is a known feature/bug (depending on who you ask) of the workflow configuration. If we retain users underlying environment variables, it makes it difficult to trace the source of the configuration.

Thanks for reporting this issue.