search

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.29k stars 1.42k forks source link

Add “trigger”-like abstraction to support push-style launches from external events #8096

Open dagsir[bot] opened 2 years ago

dagsir[bot] commented 2 years ago

Issue from the Dagster Slack

This issue was generated from the slack conversation at: https://dagster.slack.com/archives/C01U954MEER/p1653610747824519?thread_ts=1653610747.824519&cid=C01U954MEER

Conversation excerpt

U02MH8S3PPE: Triggering Automations from the Outside World

I would like to be able to trigger a job run as a result of some external event - a user clicking a button somewhere on the internet or some other system creating the event, e.g. Zapier, JotForm, Pipedream. A sensor could poll for some condition to be true (level triggering), but it would be neat to have dagster respond directly to a trigger (event triggering).

I'm aware of "Launch a run" using GraphQL (https://docs.dagster.io/concepts/dagit/graphql#launch-a-run). To use this functionality for a dagster instance running in GCP, I suppose I'd need to create a (secure) endpoint which can behave like a webhook. That endpoint would need to translate a REST API call to a GraphQL and vice versa for the response.

Anyone got recommendations for a good way to achieve this? U01GTMVMGQH: <@UM49TQ8EB> <@U016C4E5CP8> UM49TQ8EB: Hi Stefan. We’ve informally discussed adding an abstraction to be able to translate a simple action / trigger into a set of job-specific (with run config / tags) launch parameters, to support this type of “push”-style launches. I’ll add an issue so we can formally track this UM49TQ8EB: <@U018K0G2Y85> issue Add “trigger” like abstraction to support push-style launches from external events

Message from the maintainers:

Do you care about this too? Give it a :thumbsup:. We factor engagement into prioritization.

TimoFriedri commented 2 years ago

Some king of webhook based sensor would be awesome to have and open up a huge field of possible automation tasks.

E.g. in combination with any services relying on webhooks, like zapier, jotform, clickup, iftt, ... (endless list)

Even though dagster focuses on ETL and data assets, I would love to use it for these kind of automations (receive json, process, call a REST API).

chrishiste commented 1 year ago

+1

johnoscott commented 1 year ago

FiveTran has webhook support which I want to use to trigger a sensor that currently incessantly polls to check if a FiveTran job has completed. This kind of push notification would be much more efficient.