Multiple distinct graphs

[hjoliver]

This comes up fairly regularly, but we don't seem to have an issue for it.

Many (most?) workflows have "prep" tasks in the first cycle that need to finish before any future-cycle tasks begin. And similarly, there's often a need for special clean up tasks in the final cycle point, after everything else has finished.

We can do this easily enough (use prep tasks as an example) with perpetual dependence on the initial cycle point:

R1 = "prep"
P1 = "prep[^] => foo => bar"

or with artificial dependencies to force the first tasks in each cycle to start in order:

R1 = "prep => foo"
P1 = "foo[-P1]:submit => foo => bar"

However, neither of these solutions is very elegant. The first is correct in principle, but it seems a bit silly retain the ^ dependency forever, and it has performance implications for long-running workflows due to increasingly long datetime sequence computations. The second is artificial and unnecessarily prevents cycles from running out of order.

What users really want here is just a separate "start-up graph" that runs to completion before the main cycling graph starts.

Proposal A: allow multiple sequential graphs under the scheduling section

[scheduling]
   cycling mode = integer
   initial cycle point = 1
   [[graph_1]]
        R1 = "prep"
   [[graph_2]]
        P1 = "foo => bar" 
   [[graph_3]]
        R1 = "cleanup"

Proposal B: allow multiple sequential [scheduling] sections

Multiple graphs would probably suffice for most use cases, but for more power and flexibility we should have separate control of cycling, queues, etc., for each of the distinct workflow graphs.

[scheduling_1]
   [[graph]]
        R1 = "prep"

[scheduling_2]
   cycling mode = integer
   initial cycle point = 1
   [[graph]]
        P1 = "foo => bar" 

[scheduling_3]
   [[graph]]
        R1 = "cleanup"

[hjoliver]

I think this would be quite easy to implement. The scheduler only needs to manage one graph at a time. If there is nothing more to do (and we're not stalled) shut down if there are no more graphs, or start running the next graph if there is one.

[hjoliver]

(Context: in my experience, this is a common problem for new users: how do I stop cycle 2 from starting before cycle 1 is finished doing its prep work?)

[oliver-sanders]

More generally multiple distinct graphs can be considered subgraphs, this has previously been discussed in the light of #1962.

[hjoliver]

Yes I'm aware this will be solved on the Cylc 9 timescale, but that's probably several years off. I'm suggesting here that the "initial graph" and "final graph" problem can be solved quite easily under the current framework. (Thanks for the link to the wiki page though - had forgotten that).

[oliver-sanders]

but that's probably several years off

I'm a little more optimistic myself :grin:.

We might be able to knock something together under the current framework, but I'm not sure that's a good idea. This essentially opens the door to multi-dimensional cycling (only these cycles would be prisons?). This would have implications on CLI/UUID, task/job dir structure, UI / Review, etc. All of which need careful thinking through to avoid creating future problems. I would rather wait until we are able to give this the time it needs rather than rushing it in before we are ready.

how do I stop cycle 2 from starting before cycle 1 is finished doing its prep work

Odd idea, how about using the runahead limit to hold everything after the first cycle until the first cycle has completed?

runahead limit = R1, P5?

generally multiple distinct graphs can be considered subgraphs

One of the issues I see people hitting is with graphs like this:

# start_up => main_graph.every(P1D) => wrap_up

R/^ = start_up
P1D = main_graph
R/$ = wrap_up

Where each of the three sub-graphs may be dynamic. This makes it hard to connect dependencies between the different graphs as you would need to parse the graph (introspection) in order to write the appropriate start_foo[^] => main_bar dependencies.

The other way to achieve multi-dimensional cycling in Cylc being sub-workflows. If we manage to streamline these, especially from the UI perspective, they could provide a neat alternative (without the cycle as a prison limitation) for this sort of case.

[scheduling]
    [[graph]]
        R1 = start_up => main_graph => wrap_up

[runtime]
    [[start_up]]
        type = sub-workflow
        path = start-up
        template variables = True

    ...

[hjoliver]

(only these cycles would be prisons?)

True, but that's what's actually wanted for the "start up graph" problem. However, point taken that this capability could be abused horribly (if we allowed an arbitrary number of such distinct graphs)!

Odd idea, how about using the runahead limit to hold everything after the first cycle until the first cycle has completed?

Hmm, that might be good idea! An almost zero-effort trick to handle start-up graphs without exposing ourselves to other complications.

The other way to achieve multi-dimensional cycling in Cylc being sub-workflows

I like that, of course, but it may be a while again before we can do that properly.

"Start-up graphs" are something that I feel we should have supported long ago, and not hard to do, but we never got around to it.

However, your runahead idea could be a good solution in the short term.

[MetRonnie]

Imho this is more of a https://github.com/cylc/cylc-flow/labels/speculative issue than https://github.com/cylc/cylc-flow/labels/question

[hjoliver]

Well, the question was just should we do this? esp. as after a little though I think it'd be quite easy to do. It's not particularly speculative, as the need for start-up and shutdown-graphs has come up multiple times over the years.

The multi-dimensional cycling, but the cycles are "a prison" thing is really not an issue for my intended use case, because that is literally what's wanted for start-up and shutdown graphs. However, for more general multi-dimensional cycling it certainly is an issue, and my proposal could be abused in that way (albeit to a limited extent if we didn't allow an arbitrary number of separate graphs).

However, I'm quite happy to go with @oliver-sanders variable runahead as an interim solution .That will be super easy to implement and would handle the primary use case (start-up graph) very nicely.

So I'll close this, and open a new issue for that...