Step invoked by output_from/named_output/provides cannot recognized task variable defined in global statement.

[hsun3163]

Hi Bo,

I have encountered an error where invoking other step by output_from , named_output , and provides will cause the task statement not being able to recognize the global statement. Following error was given

ERROR: [A (A)]: [A]: Failed to execute process
"bash(fr"""touch {_output}\n""", stderr = f'{_output}.stderr',...'{_output}.stdout')\n"
name 'walltime' is not defined
[B]: Exits with 1 pending step (B)

when running

sos run test_global.sos B -J 1 -q csg -c csg.yml

Where test_global.sos is:

[global]
# The output directory for generated files. MUST BE FULL PATH
parameter: cwd = path("./")
# For cluster jobs, number commands to run per job
parameter: job_size = 1
# Wall clock time expected
parameter: walltime = "5h"
# Memory expected
parameter: mem = "16G"
# Number of threads
parameter: numThreads = 8

[A]
output: file = f'{cwd:a}/test_file'
task: trunk_workers = 1, walltime = walltime, mem = mem, cores = numThreads, tags = f'{step_name}_{_output:bn}'
bash: expand= "$[ ]", stderr = f'{_output}.stderr', stdout = f'{_output}.stdout'
    touch $[_output]
[B]
input: output_from("A")
task: trunk_workers = 1, walltime = walltime, mem = mem, cores = numThreads, tags = f'{step_name}_{_output:bn}'
bash: expand= "$[ ]", stderr = f'{_output}.stderr', stdout = f'{_output}.stdout'
    touch $[_input]

However, I was under the impression that this mechanism should work fine previously.

The sos version I am using is the latest. And the pbs version is sos-pbs 0.20.8

[BoPeng]

Simplifying the test case to

[global]
parameter: mem = "16G"

[A]
output: 'test_file'
task: mem = mem
bash: expand=True
    touch {_output}

[B]
input: output_from("A")
bash:
    touch {_input}.processed

with command error being

> sos run test_task_params.sos B  -s force -q localhost
INFO: Running A:
ERROR: [A (A)]: [A]: Failed to execute process
"'bash(fr"""touch {_output}\n\n""")\n'"
name 'mem' is not defined
[B]: Exits with 1 pending step (B)

Note that the pipeline works without -q localhost.

[BoPeng]

Could you test if branch issue1457 works?

[hsun3163]

Could you test if branch issue1457 works?

I am not sure how to install this branch into my cluster. I have been installing sos via pip.

[gaow]

@hsun3163 you clone that branch to your computer, cd to it and install via pip install .

[hsun3163]

I can successfully run the new testing example you created. But when I try to run my original example, or any other notebook with task statement. I recieved the following error:

ERROR: Failed to parse statement: trunk_workers = 1, walltime = walltime, mem = mem, cores = numThreads, tags = f'{step_name}_{_output:bn}'
 in exec mode

[BoPeng]

Just pushed another commit. Please try again.

[hsun3163]

I works now, thank you!

[BoPeng]

Great. I will create a new release of sos after more testing.

[gaow]

@BoPeng shall we make a new release so others will benefit from the fix? Currently this works well on our end.

[BoPeng]

ok. I have just released sos 0.22.7. I have tried to make all sos tests run in the past few days. The problem was that the tests were using subprocess-based sos runs so all file handles will be reclaimed even if sos run failed. I have changed most of the tests to execute_workflow so many tests fail due to an insufficient number of file handlers, apparently caused by (intentionally) failed tests. The entire execution step is complicated so I have not figured out a good way to clean up properly.