pool: check prereqs on task spawn

[oliver-sanders]

When we spawn a task, we currently check the DB to retrieve previous submissions. This is required for:

• Flow merge detection.
• Task re-run prevention (within the same flow).
• Preventing the accidental reincarnation of removed tasks.

Whilst we do check for previous submissions, we do not presently check task prerequisites. In some situations, task prerequisites which are satisfied in the DB, are left unsatisfied in the task pool when tasks are spawned. This can happen when tasks are added to the pool via means other than natural task satisfaction (e.g. https://github.com/cylc/cylc-flow/issues/5952).

Easy way to replicate this:

• Empty the task pool.
• Put the tasks back.
• Run the workflow on.

Sadly, I don't this it is sufficient to check the DB only when such interventions are performed, because tasks downstream of the ones being added which are yet to be spawned may also be missing this state and end up with (erroneously) partially satisfied prerequisites.

This is a bad bug as it makes it look to the user like tasks which have run, haven't. It's hard to explain, especially as multiple Cylc interfaces will provide erroneous information. Moreover it's very hard to recover from as the consequences may last as long as the longest inter-cycle dependency in the workflow.

There are efficiency concerns over requesting task prerequisites, however, we haven't ascertained that this would be a problem as yet, however, DB processing is definitely a bottleneck. One way to reduce these overheads might be to only request satisfied prerequisites from the DB. Merging this request in with the existing DB request would also improve performance. As would batching the requests where multiple tasks are spawned in the same main loop iteration.

This issue has relevance to workflow extension use cases (#5952), graph changes either by reload or restart

[hjoliver]

Sadly, I don't this it is sufficient to check the DB only when such interventions are performed, because tasks downstream of the ones being added which are yet to be spawned may also be missing this state and end up with (erroneously) partially satisfied prerequisites

Good point. It would be sufficient to check the DB only when the workflow stalls, but with significant cons:

• there could be a series of stalls until full recovery
• there could be a confusing period before the stall where tasks appear to be unsatisfied

[oliver-sanders]

Yes, this would be highly confusing.

I think the only way around this without checking prereqs for each spawn (not that I'm suggesting it per-se) would be to load in the task history from the DB and spawn on all outputs. If tasks already ran, nothing would happen, if they haven't then they will be spawned allowing the workflow to continue without the risk of artificial stall.

However, this would have the caveat that we would be spawning according to the current graph which may differ from the graph historical tasks were running according to (i.e. it could result in spawning errors).