search

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
10.71k stars 1.33k forks source link

Support Kafka Consumer message queue to handle streaming pipeline #11857

Open ailabx opened 1 year ago

ailabx commented 1 year ago

What's the use case?

can Dagster support kafka to consume message queue， handle a streaming pipline?

Ideas of implementation

just like airflow or prefect?

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

feizeikesi commented 1 year ago

i think it's a good idea👍🏻

cmpadden commented 1 month ago

@ailabx we're giving some thought on how we can best support this natively, however, for the time being, you can reference this example project for ingesting Kafka topics via sensors.

https://github.com/cmpadden/dagster-kafka-demo/blob/main/dagster_kafka_demo/sensors.py