oliver-sanders
commented
1 month ago task is incomplete if its outputs are incomplete, whether or not it has finished
As per https://github.com/cylc/cylc-admin/pull/191 (and related discussion), there are a couple of issues with using "finished" in this way:
- It's not technically correct as "finished" infers that the task has started which incorrectly excludes "submit-failed" and "expired".
- It conflicts with the ":finished" qualifier which is correctly defined as succeeded or failed.
Suggest continuing with "task with final status" or simply "final task" for this purpose.
a stall occurs if there is nothing left to run but there are any incomplete tasks in the n=0 window
Unfortunately, there problems with this statement:
- Now, "completion" is tied entirely to "output completion", the phrase "incomplete tasks" includes active tasks. So this definition infers that a workflow can stall whilst tasks are still queued, preparing, submitted or running which is incorrect.
- Tasks with unsatisfied prerequites may actually have complete outputs (it is possible for a task not to have any required outputs at all) so "incomplete task" does not cover "task with partially satisfied prerequisites".
- Tasks waiting on xtriggers or extriggers may have incomplete outputs, but should not stall a workflow.
So "incomplete tasks" are not sufficient to define workflow stall or completion without also involving final task status.
(There are also a couple of small omissions, due to held tasks and the paused workflow status).
Now, "completion" is entirely to "output completion". That means it is legit to say that a task is incomplete if its outputs are incomplete
We are ok with brining back the phrase "incomplete task" in this manner (there's no issue with the term or definition), however, the term isn't especially useful as we usually simply refer to them as "waiting tasks". I'm not sure we would find uses for this new term other than in combination with "final task status".
The much more important term is "incomplete task with a final status" for which you've used, but not defined "finished incomplete task" (which has the issues outlined above). We still prefer the original definition of "incomplete task" (including status), but failing that (ground already conceded), could we have "final incomplete task" as a defined term? We need to refer to this a lot, we also need a task event for it (https://github.com/cylc/cylc-flow/issues/4957), so could really do with a defined name or phrase.
hjoliver
A quick blast through the docs (mostly the glossary) whilst trying not to overlap too much with the optional-outputs-extension, which should be documented separately (mostly by pasting from the proposal).
(I did add a glossary item on output completion, needed for cross-referencing).
note on "incomplete"
We agreed not to define an "incomplete task" as a finished task with incomplete outputs (because that made "incomplete" different to "not complete" ).
Now, "completion" is tied entirely to "output completion". That means it is legit to say that a task is incomplete if its outputs are incomplete, whether or not it has finished. That's convenient and I've used it a couple of times here. E.g. it means we can say a stall occurs if there is nothing left to run but *there are any incomplete tasks in the n=0 window" - that covers both finished-with-incomplete-outputs and waiting-with-partially-satisfied-prerequisites. Both of those are "active tasks" (i.e., in n=0) and have incomplete outputs.
Requirements check-list
CONTRIBUTING.mdand added my name as a Code Contributor.