Support for Kafka - Githubissues

spotify / scio

A Scala API for Apache Beam and Google Cloud Dataflow.

https://spotify.github.io/scio

Apache License 2.0

2.55k stars 512 forks source link

Open gmjohnsonjh opened 2 months ago

gmjohnsonjh commented 2 months ago

Scio has support for the GCP product pubsub. Was support for Kafka ever considered? Were there any known challenges in the implementation?

RustedBones commented 2 months ago

Hi! thanks for the feature request. We can indeed create a scio-kafka module that wraps beam's KafkaIO.

To my knowledge, there isn't any particular challenges around this IO. Are you interested at giving it a try ?