numaproj / numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs
https://numaflow.numaproj.io/
Apache License 2.0
1.09k stars 111 forks source link

Support for mulitiple kafka topics by regexp #1761

Open th0ger opened 3 months ago

th0ger commented 3 months ago

Summary

Some kafka clients allows regexp patterns for the topic name to consume from multiple topics.

Is this possible with the sarama client? Or can this be supported?

Here from the Confluent kafka python Consumer:

subscribe(topics[, on_assign=None][, on_revoke=None][, on_lost=None]) Set subscription to supplied list of topics. (...) Regexp pattern subscriptions are supported by prefixing the topic string with "^", e.g.: consumer.subscribe(["^my_topic.*", "^another[0-9]-?[a-z]+$", "not_a_regex"])

Parameters topics (list(str)) – List of topics (strings) to subscribe to. (...)


Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.