external event triggers

[hjoliver]

We could allow tasks to trigger off external events completely generically - without having to support OS-specific things like inotify - by having task prerequisites that are not associated with another task, so they'd have to be satisfied by a cylc CLI call (or potentially, cylc internal API). This could be used to initiate some action in the suite without requiring a suite task that repeatedly polls for the external event (or using automatic task retries to do the polling). Sometimes polling may not be so convenient, e.g. to wait on an external dataset that arrives randomly and very infrequently, or to wait on a large number of external files individually.

This is essentially how message triggers work already, except that they have to come from another task. So we could already bodge this in with a dummy task that depends on message triggers (with messages emitted by an external system that, say, watches for a file). [credit: Joan Fernon at BoM].

So I propose formally supporting external event triggers, something like this:

graph =  <external_x>:event_1 & <external_x>:event_2 => foo
[runtime]
   [[external_x]]
      # (graph syntax identifies "external_1" as an event rather than a task)
      [[[outputs]]]
         event_1 = "dataset 1 arrived"
         event_2 = "dataset 2 arrived"

This would make foo depend on two messages as defined, coming from an external event that self-identifies as "external_x". But no task called "external_x" would be created in the suite.

@matthewrmshin et. al. - thoughts?

[hjoliver]

But no task called "external_1" would be created in the suite

A proxy object would still be created, to receive the event messages.

Some thought required on how to handle multiple instances of an event proxy in a cycling suite, if the event doesn't have an associated cycle point itself (thinking of the satellite data processing example).

[matthewrmshin]

Idea is good. (As usual, I'd expect some debate on the actual user interface + implementation. :wink:)

[hjoliver]

NIWA's satellite data processing will convert to cylc shortly. External event trigger seems to be a better way of handling this sort of thing because we can have data retrieval tasks that trigger when needed, instead of continuously polling at run time, or continually retrying until new data arrives. As a result, I thought through this issue today and banged up an initial implementation. I'll put up a pull request for consideration shortly.

[matthewrmshin]

The use case you have just described is quite similar to something our users would like to do. I'll look forward to the pull request.

[hjoliver]

I've revised the initial proposal above. Here's the reasoning that led to my prototype implementation:

We need a repeating workflow to handle a succession of arbitrarily timed datasets in real time:

• In the cylc tradition, we should be able to process multiple datasets in parallel (they could come in quickly, or the suite could be restarted after multiple sets have come in)
• Use integer cycling to get repetition without date-time cycle points
• The integer cycle point is arbitrary, it just serves to differentiate between successive workflows
• Task proxies can register external triggers such as "new sat-x dataset ready for processing" that need to be satisfied before they can submit to run. Unlike normal task dependencies, these strings cannot contain "current (or offset) cycle point" because the external system that actuates the trigger cannot be expected to know the arbitrary cycle point of the task instance currently waiting on the trigger. In fact the minimum targeting information needed is the suite name (and host), so suite daemons need a single object that listens for all external trigger events - I'll call this the external trigger broker.
• For a particular external trigger, there is nothing to distinguish one event from the next (due to lack of a cycle point that is meaningful outside of the suite daemon)
  • (actually a unique arbitrary ID - such as the filename of the new dataset or its wall-clock arrival time - can be supplied with each trigger event, but the exact form of this ID cannot be known in advance, and besides this is not strictly necessary because the mapping between dataset and cycle point is arbitrary: the only thing that matters is that tasks in a cycle point (any one cycle point) process a dataset (any dataset that has already come in)).
• The external trigger broker will queue incoming (and typically identical) external trigger messages, and will provide them for the purpose of satisfying the registered external triggers of task proxies.
• Because successive trigger messages are identical, they will have to be discarded immediately once used to prevent them from erroneously triggering tasks again in the next cycle. This in turn - because some tasks spawn into the next cycle sooner than others - means only one task can register a particular external trigger. This will need to be documented, but it can be enforced during suite validation, and it is not a restriction in any practical sense because multiple tasks can trigger (in the normal sense) off one task with an external trigger.
• If a dataset ID is optionally supplied with each trigger event, the ID can be broadcast to all tasks with the same cycle point, so that they can call distinguish the current/new dataset from other datasets. If an ID is not supplied, however, the data retrieval tasks just need to be designed to move each new dataset to a shared cycle-point-specific work directory, then the downstream tasks can simply process the files that they find in their share space - in effect the data retrieval task associates the dataset with its cycle point.

Finally, in contrast to my original proposal above, external triggers should not appear as nodes in the graph because (a) they are not task proxies; and (b) they are essentially very similar to clock-triggers (we might eventually like to annotate graph nodes to indicate the presence of clock- and external triggers). [Update: opinion changed on this: https://github.com/cylc/cylc/issues/1364]