Is it possible to have source from more than one topic?

digitalocean / firebolt

Golang framework for streaming ETL, observability data pipeline, and event processing apps

Other

702 stars 50 forks source link

Is it possible to have source from more than one topic? #29

Open marshallharry opened 3 years ago

marshallharry commented 3 years ago

Its not really an issue but I'm wondering if is it possible to have multiple kafka topic in the source with the same node processing the message? Thank you.

marshallharry commented 3 years ago

@jnadler @cristovarghese please advice, sir

jnadler commented 3 years ago

No, multiple topics are not supported today. However this would not be too hard to add to firebolt, because confluent-kafka-go natively supports subscribing to multiple topics at once. It would involve changes in kafkaconsumer.go and recoveryconsumer.go at a minimum. Perhaps we'd expect a CSV string of topics, and split the string into the array used to configure the consumers.