Event management use cases in Kubernetes

[edlee2121]

Let's use this issue to collect use cases for event management in Kubernetes.

The original motivation for Argo Events is to have a very light-weight, Kubernetes-native mechanism for automatically triggering Argo workflows in response to external and internal events.

Use cases:

• Trigger actions based on Kubernetes events
  • Use it like Lambda for Kubernetes system/application management
• Trigger actions based on external events such as
  • New object created in an S3 bucket
  • Messages (e.g. SNS/MQTT/AMQP)
  • GitHub commit/PR (for CI/CD)
• Trigger actions based on schedule/time
• Trigger workflows for ML/analytics/data-processing applications

What should the Argo Events experience be?

• Kubernetes-native (like Argo workflows)
  • Based on declarative specs
• Lightweight, low operational overhead
• Extensible
  • New event sources

What are we NOT trying to do with Argo Events?

• Replacement for general purpose messaging or pub/sub systems like ActiveMQ or Kafka.
• Replacement for FaaS systems like Lambda, Serverless, Kubeless, etc.
  • Very high event rates.
  • Simple functions.

Requirements/Open Issues

• Persistence/reliability
  • At least once semantics
• Target performance
  • 10K events per second
• Level vs. edge-triggered events
• Use CloudEvents for describing events (https://github.com/cloudevents/spec)?
• Allow actions to be conditioned on more than one event?
• Security
  • Who has access to what events?
  • What is the scope of the triggered actions?

Related classes of systems

• Google Cloud Functions, Lambda
• Kubeless, Fission, Serverless
• Brigade (event handling + scripting)
• Azure Event Grid, Serverless Event Gateway

[slongay]

Maybe for internal events too, like a Kafka message?

[magaldima]

• Trigger events based on a schedule or time
• Trigger events based on messages from an MQTT or AMQP queue

Just some thoughts from my initial work: an S3 bucket trigger is really just an SNS or SQS notification, how tightly coupled the bucket-notification to the actual trigger is probably depends on how often that bucket is used and/or shared.

[edlee2121]

@magaldima Thanks, I updated the original comment. @slongay What do you mean when you refer to Kafka messages as an internal event?

[vicaire]

Would the following be a good fit for "argo-events"?

• Triggering an Argo workflow when another Argo workflow finishes? (Would this be covered by "Trigger actions based on Kubernetes events"?)

• Should "argo-events" support triggering more K8 resources than just workflows? For instance, say I just want to run a single container instead of an Argo workflow, would "argo-events" support this?

• Would "argo-events" provides support for "data-centric workflows"? Let me provide an example: each time new data is added at location L1, it triggers the execution of a container/workflow that processes the data in L1 and stores the results to L2, which triggers the execution of a container or workflow that processes the data in L2 and stores the results to L3, etc. Would "argo-events" support specifying such a "data-centric" workflow (based on "there-is-new-data" events) as a single K8 resource?

[edlee2121]

@vicaire Great examples of what we could do with Argo events!

1) Yes, while an Argo workflow could manually trigger another workflow, it could be more elegant to string together workflows at the event management level. 2) Yes, I think we should allow actions to run a pod or perhaps invoke more general k8s api actions. 3) Yes, this is our intent. I like your description of how Argo events could be used to run data flow applications, where users would not even need to be aware that Argo events or even Kubernetes is the underlying platform.

The possibilities seem endless. We will need to be careful in choosing the right set of use cases for the initial design.

[wookasz]

Interesting project which addresses use case mentioned during community meeting of triggering workflows based on data creation https://github.com/pachyderm/pachyderm

[wookasz]

@edlee2121 not sure if this is what @slongay had in mind, but the ability to trigger workflows from Kafka messages would be great (saying this selfishly as 99.9% of our backend service communication is over Kafka). I envision something like a mapping of Kafka topic to workflow. argo-events would subscribe to each topic and kick off the corresponding workflow passing the topic parameters and payload as workflow parameters.

[edlee2121]

@wookasz really interesting idea! It should be straightforward to create a Kafka consumer group as a deployment that reads from a specified set of topics, perhaps does some filtering, and then creates an Argo event.

[slongay]

@wookasz Yup exactly. I have been playing around using a simple serverless function (kubeless) subscribed to kafka events which trigger argo. It seems like a promising workflow so far.

[elibixby]

I'd like to be able to run argo workflows that continually produce events, and run other argo workflows that consume these continuous streams of events (various ML usecases for this), without needing to be reloaded (either on the triggering or consuming side).

[edlee2121]

@elibixby Do you mind describing your use cases in more detail? When you say reloaded, is that the same as starting a new workflow? Why don't you want the workflow reloaded? Do they maintain a lot of in memory state? Thanks.

[pplavetzki]

I really like the way Brigade allows you to build custom gateways that will take external events i.e. kafka and amqp messages etc. and turn them into events that can be published into the framework. This would allow you to develop a common interface that must be implemented to inject events into argo.

I have an application that leverages OpenStack's RabbitMQ to trigger individual containers to process the OpenStack event. If i could use that same event to inject it into Argo to run a workflow, I'd be ecstatic!

[edlee2121]

@pplavetzki We definitely want to make it easy to inject all types of events into Kubernetes.

[elibixby]

@edlee2121 Yes, lots of state in memory. Ideally an argo container could produce "outputs" as normal and these outputs would be considered a stream of events. Then downstream tasks in the workflow would be triggered based on some windowing policy over these events, receiving the "window" as inputs.

[edlee2121]

@elibixby One way to handle this may be to delay triggering events until certain batching or timing criteria are satisfied.

[elibixby]

@edlee2121 this has very poor performance implications, since delaying applications of trajectories to RL policies means the next time an action needs to be selected, delayed trajectories must be applied. See this paper for an example of the performance improvements from making this asynchronous: https://arxiv.org/abs/1802.01561

I have some thoughts on how I'd like to see events integrated into Argo as a first class citizen, is this the best place to put them?

[edlee2121]

@elibixby Yes, please share additional details about your use case and how you would go about addressing them.

[elibixby]

@edlee2121 I have a rather long proposal written up. Normally I would submit this as a PR with a MD file so people can comment on individual lines, but this repo doesn't have any files yet, so I can't fork it to create a PR. Would you prefer I submit it as an issue on this repo, or a PR on argoproj/argo (it does relate to the core workflow orchestration system, since I think events need to be added as a first class citizen of the core orchestration system to be their most useful).

Note: I think the proposal meets all of the requirements of people in this thread so far, so it's not restricted to my use cases.

[edlee2121]

@elibixby I pushed a README. Go ahead and submit your PR.

[elibixby]

@edlee2121 Thanks! I submitted #4 for comment! It's rough, but contains a lot of thoughts that are hard to clarify in short-form comments.