ecmwf / pyflow

A high level Python interface to ecFlow allowing the creation of ecFlow suites in a modular and "pythonic" way
https://pyflow-workflow-generator.readthedocs.io/en/latest/
Apache License 2.0
7 stars 7 forks source link

exit_hook in Task not scanned for environment variables #32

Open lotruheawea opened 9 months ago

lotruheawea commented 9 months ago

What happened?

When a user defines a custom exit_hook for a given Task, contained ecFlow variables are not exported to environment variables: I guess the library needs to be adjusted in nodes.py line 1321 from all_scripts = "\n".join(script + workdir_lines + module_lines) to all_scripts = "\n".join(script + workdir_lines + module_lines + self._exit_hook)

What are the steps to reproduce the bug?

import pyflow as pf

with pf.Suite('hello_pyflow',
              host=pf.SLURMHost('foo'),
              files='/home/username/tmp/',
              MY_VAR='bar',
) as s:
    pf.Label('greeting', '')
    pf.Edit(HOOK_VAR='foo')
    t1 = pf.Task('t1', script='echo "I am on $(hostname) : $ECF_HOST, $MY_VAR"', exit_hook='echo ${HOOK_VAR}')
s.deploy_suite(overwrite=True)

Version

v3.1.0

Platform (OS and architecture)

Linux ac6-101.bullx 4.18.0-372.52.1.el8_6.x86_64 #1 SMP Fri Mar 31 06:22:44 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

Relevant log output

No response

Accompanying data

No response

Organisation

No response

corentincarton commented 6 months ago

Hi @lotruheawea, sorry for the late answer. As the exit hook is not part of the core script, so you should be able to use ecflow variables directly there: exit_hook='echo %HOOK_VAR%'. Exporting the ecflow variable wouldn't help as they are exported after the definition of the exit hook.