search

tremor-rs / tremor-runtime

Main Tremor Project Rust Codebase
https://www.tremor.rs
Apache License 2.0
851 stars 126 forks source link

Add support for selecting topic dynamically for a Kafka offramp #1296

Open anuprout opened 2 years ago

anuprout commented 2 years ago

Describe the problem you are trying to solve Add support for selecting topic dynamically for a Kafka offramp. Right now the topic is set based on the offramp configuration. However this can be passed at run time for each event.

Describe the solution you'd like Enhance the Kafka offramp to override the topic name based on metadata (lets say $topic) associated with an event.

Licenser commented 2 years ago

We're trying to standardize the way connectors handle this kind of non data events. KV is right now the template and it uses metadata for issuing commands "get/put/scan/delete/cas". Would it make sense to add metadata capabilities to the kafak connectors?

Subscripbe / unsubscribe / change topics for events?

anuprout commented 2 years ago

Adding this feature to Kafka connector would be nice. Thanks