search

argoproj / argo-events

Event-driven Automation Framework for Kubernetes
https://argoproj.github.io/argo-events/
Apache License 2.0
2.37k stars 738 forks source link

Query-based data gateway #207

Closed magaldima closed 4 years ago

magaldima commented 5 years ago

Is your feature request related to a problem? Please describe. Currently, argo-events only supports data-driven event pipelines for S3 like stores. This is a proposal to build a gateway to support watching/registering for events from an SQL or query-based backend data store.

Describe the solution you'd like A new gateway implementation that makes a connection to a database, supports a query pattern, and implements a simple cron expression to manual poll (or query in this case). We can probably leverage the go-cloud project's ability to connect to MySQL or PostgreSQL DBs for now. I believe we'd have to be opinionated about the conditions for satisfying an event and also the payload for which the event contains.

Describe alternatives you've considered I'd also be interested in looking at databases where you don't have to implement manual polling mechanisms and the DB itself has a concept of triggers and possibly integrating argo-events to be a subscriber of these events...

Additional context Add any other context or screenshots about the feature request here.

VaibhavPage commented 5 years ago

This is interesting. Can you elaborate more on I believe we'd have to be opinionated about the conditions for satisfying an event and also the payload for which the event contains. ?

magaldima commented 5 years ago

Event conditions - I mean like a structured API or interface for performing the actual query / search. This has implications around what DBs/languages to support. A simple example would be to support SQL - like queries through an already established interface like the golang db package. This could be even easier to implement by leveraging the go-cloud project. The difficult part is to create conditionals out of query results.. do you trigger an event simply based on non-nil results?

Payload - what exactly does the payload for the event contain in this instance? The results/rows of the query?

tooptoop4 commented 2 years ago

can this be re-opened @VaibhavPage ? some of https://github.com/argoproj-labs/argo-dataflow/pull/215/files might be re-usable