nipype / pydra

Pydra Dataflow Engine
https://nipype.github.io/pydra/
Other
119 stars 57 forks source link

Workflow hangs splitting 3-times and then combining once #668

Open tclose opened 1 year ago

tclose commented 1 year ago

What version of Pydra are you using?

master branch

What were you trying to do?

Run the following workflow

from pydra import mark, Workflow

@mark.task
def identity(x):
    return x

wf = Workflow(name="myworkflow", input_spec=["x"], x=1)
wf.add(identity(name="a").split("x", x=wf.lzin.x))
wf.add(identity(name="b").split("x", x=wf.a.lzout.out))
wf.add(identity(name="c").split("x", x=wf.b.lzout.out))
wf.add(identity(name="d", x=wf.c.lzout.out).combine(["a.x"]))
wf.add(identity(name="e", x=wf.d.lzout.out))
wf.set_output(("out", wf.e.lzout.out))

wf.inputs.x = [[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]]

result = wf(plugin="cf")
print(result.output.out)

What did you expect will happen?

The workflow would complete successfully

What actually happened?

The loop at

https://github.com/nipype/pydra/blob/426564eca3efa7d840fdca33bdbef6a1c88e2ffc/pydra/engine/submitter.py#L165-L166

hangs, due to get_runnable_tasks not finding a complete set of results.

It appears the problem relates to the final_combined_ind_mapping attribute having empty values here

https://github.com/nipype/pydra/blob/426564eca3efa7d840fdca33bdbef6a1c88e2ffc/pydra/engine/core.py#L697-L701

Can you replicate the behavior? If yes, how?

Yes, by running the example workflow above

tclose commented 1 year ago

Note that not combining and combining over 2 or 3 of the splits runs