cylc / cylc-doc

Documentation (User Guide, Cheat Sheets, etc.) for the Cylc Workflow Engine.
https://cylc.github.io/cylc-doc/
GNU General Public License v3.0
8 stars 18 forks source link

interventions: setting off-flow prereqs #745

Open oliver-sanders opened 2 weeks ago

oliver-sanders commented 2 weeks ago

We have documented how to re-run a chain of tasks using --flow=new.

https://cylc.github.io/cylc-doc/stable/html/user-guide/interventions/index.html#re-run-multiple-tasks

This example is great, but it doesn't show you what to do if your sub-graph ain't a chain.


Context...

For example, take this workflow:

[scheduler]
    allow implicit tasks = True

[scheduling]
    [[graph]]
        R1 = """
            a => b => c => d
            x => b
        """

[runtime]
    [[d]]
        script = false

If we run this workflow, it will get to task "d", which will fail causing the workflow to stall, leaving this message in the log:

ERROR - Incomplete tasks:
* 1/d did not complete required outputs: ['succeeded']

Let's say we need to make a change to the task "a" and re-run everything after it to recover. We would trigger "a" in a new flow like so:

$ cylc trigger //1/a --flow=new

This starts a new flow, however, the workflow quickly stalls again because the task "1/x" hasn't run in the new flow, resulting in this message in the workflow log:

WARNING - Partially satisfied prerequisites:
* 1/b is waiting on ['1/x:succeeded']

To solve this, we tell Cylc that "1/x" has already succeeded, effectively telling Cylc to pull in this task from the previous flow:

$ cylc set-outputs //1/x

After this, the workflow runs on as expected:

INFO - [1/x waiting(runahead) job:00 flows:none] Forced spawning on succeeded
INFO - [1/b waiting job:01 flows:2] => waiting(queued)

It's the "set-outputs" (now just "set") part of this process which isn't presently explained.

Suggest updating the example, or adding a new one, which shows a graph like the one above and explains that the "off-flow prerequisite" either needs to be re-run (trigger) or ignored (set) and shows how to do it.