PROPOSAL: Events as first class objects in pipelines

[elibixby]

This proposal is the result of a lot of discussion around how to build "data-driven pipelines". The semantics of Argo workflows are still useful for an event based system, and rather than build a separate workflow system, I propose adding event semantics to the existing workflow system.

All of this is conceptual. But I wanted everyone to at least explore the idea that a DAG orchestration system can benefit from events, and an event system can benefit from DAG workflows.

Some areas I'd especially appreciate some input:

• Where should windowing/filtering policies be defined? Should we make separate tasks that aggregate and filter events and re-emit as "window" events. Should the workflow definition have "list" like syntax for inputs (i.e. {{steps.foo.outputs.my-event[0]}}) and how does that map to unbounded streams?
• What should the configuration of "eventPolicy" look like. This field is supposed to be "how do events effect this task" so you can imagine a large number of possible values, but most of these can be emulated by adding more event inputs. For example, instead of having a "create_synchronous" which requires at most one task of that type is running at a time, and a "create_asynchronous" you can have the downstream task emit a lifecycle event, and only trigger if the lifecycle status of the event stream is not running.

[edlee2121]

Thank you for writing this up. I may not be reading it right but it reminds of a streams type of abstraction where the workflow is setting up steps that communicate via an event stream.

FYI, BlockRock just contributed their implementation of events to the Argo Project (https://github.com/argoproj/argo-events). Please check it out and continue the discussion on the argo-events slack channel (https://argoproj.slack.com/messages/CAM37QYMS)