DataBiosphere / toil

A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.
http://toil.ucsc-cgl.org/.
Apache License 2.0
900 stars 240 forks source link

WDL input expressions cannot reference output of tasks #5038

Closed stxue1 closed 3 months ago

stxue1 commented 3 months ago

The WDL spec says that the order of declarations can be arbitrary: https://github.com/openwdl/wdl/blob/16267770d12f28706d0d10e000700887f3649970/SPEC.md?plain=1#L5124

The referenced unit test does not pass Toil: https://github.com/openwdl/wdl/blob/16267770d12f28706d0d10e000700887f3649970/SPEC.md?plain=1#L5128-L5158

This is because we evaluate sections sequentially, with the input section of a workflow evaluated before the task is launched:

[2024-07-24T20:08:17-0700] [MainThread] [E] [toil.wdl.wdltoil] Expression evaluation failed for y: d1.out
    Traceback (most recent call last):
      File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/WDL/Expr.py", line 124, in eval
        ans = self._eval(env, stdlib)
              ^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/WDL/Expr.py", line 864, in _eval
        return env[self.name]
               ~~~^^^^^^^^^^^
      File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/WDL/Env.py", line 127, in __getitem__
        return self.resolve(name)
               ^^^^^^^^^^^^^^^^^^
      File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/WDL/Env.py", line 114, in resolve
        return self.resolve_binding(name).value
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/WDL/Env.py", line 106, in resolve_binding
        raise KeyError()
    KeyError

Thus there is no task to lookup.

┆Issue is synchronized with this Jira Story ┆Issue Number: TOIL-1622

stxue1 commented 3 months ago

The next unit test is also dependent on this behavior: https://github.com/openwdl/wdl/blob/16267770d12f28706d0d10e000700887f3649970/SPEC.md?plain=1#L5200-L5220

adamnovak commented 3 months ago

Isn't this a duplicate of #4993?

stxue1 commented 3 months ago

Oops