cylc / cylc-flow

Cylc: a workflow engine for cycling systems.
https://cylc.github.io
GNU General Public License v3.0
333 stars 94 forks source link

skip mode: consider using special "skip" task messages #6380

Open oliver-sanders opened 1 month ago

oliver-sanders commented 1 month ago

Extension to https://github.com/cylc/cylc-flow/pull/6039 and https://github.com/cylc/cylc-admin/blob/master/docs/proposal-skip-mode.md#proposal-skip-run-mode-for-tasks

Note: This applies to all "non-live" run modes (i.e. dummy, skip and simulation), for the purpose of this issue I've focussed on skip.

Task outputs in the DB

When tasks are run, we store any messages and outputs they generated into the task_outputs table in the database.

E.G:

[scheduling]
    [[graph]]
        R1 = foo

[runtime]
    [[foo]]
        script = cylc message -- my-message
        [[[outputs]]]
            my-trigger = my-message
        [[[skip]]]
            outputs = my-trigger
$ cylc vip -N .
$ sqlite3 ~/cylc-run/tmp.PQ7h6sdpgP/runN/.service/db 'SELECT * FROM task_outputs;'
1|foo|[1]|{"submitted": "submitted", "started": "started", "succeeded": "succeeded", "my-trigger": "my-message"}

The custom trigger "my-trigger" gets stored along with the "my-message" task message that generated it.

Skip mode messages

Skip mode is no different, we still have to store the task outputs, however, the task messages don't make much sense here as they were internally generated.

E.G. The message "my-message" shouldn't really be in the DB if the task was run in skip mode because it doesn't make sense.

Skip mode prerequisites

In the proposal we agreed on this:

When tasks are run in skip mode, the prerequisites which correspond to the outputs they generate should be marked as satisfied by skip mode rather than satisfied naturally for provenance reasons.

For the purpose of cylc remove logic, satisfied by skip mode should be treated the same as satisfied naturally.

Proposal / Question

Extend this prerequisite logic to cover task messages.

I.E. The task message for any output satisfied by skip mode should be recorded as "satisfied by skip mode".

$ cylc vip -N .
$ sqlite3 ~/cylc-run/tmp.PQ7h6sdpgP/runN/.service/db 'SELECT * FROM task_outputs;'
1|foo|[1]|{"submitted": "satisfied by skip mode", "started": "satisfied by skip mode", "satisfied by skip mode": "succeeded", "my-trigger": "satisfied by skip mode"}

This would also extend to the other non-live modes (dummy & simulation).

This change would be visible in the GUI:

Screenshot from 2024-09-24 16-08-39

oliver-sanders commented 1 week ago

Meeting 2024-11-04

Agreed, removing the question label.