search

digitalocean / firebolt

Golang framework for streaming ETL, observability data pipeline, and event processing apps
Other
698 stars 50 forks source link

Feature/multiple topics #30

Closed cristovarghese closed 3 years ago

cristovarghese commented 3 years ago

Support for multiple kafka topics in firebolt/kafkaconsumer. Fixed existing test cases as well. To maintain compatibility with old firebolt config file, "topic" param under kafkaconsumer sources can now be a comma separated list of kafka topics.

jnadler commented 3 years ago

Hi @cristovarghese ! Did you intend to close this PR? I'd be happy to help get this across the finish line, let me know if there's anything I can do.

cristovarghese commented 3 years ago

Hi Jeff,

Yes on further testing I noticed there was a bug when multiple queues were configured. When firebolt restarts the same message are being delivered again(which were already processed) from the queue. With a single queue there are no issues.

Regards,

Christo @.***

On Fri, 9 Jul 2021, 11:04 pm Jeff Nadler, @.***> wrote:

Hi @cristovarghese https://github.com/cristovarghese ! Did you intend to close this PR? I'd be happy to help get this across the finish line, let me know if there's anything I can do.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/digitalocean/firebolt/pull/30#issuecomment-877346625, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACY3CLXLM4XR5UVIY4HF24DTW4XLJANCNFSM5AC76PNA .