search

peterprib / node-red-contrib-kafka-manager

Implement
GNU General Public License v3.0
22 stars 12 forks source link

Wildcards on topic #30

Closed janainascal closed 3 years ago

janainascal commented 3 years ago

Hi @peterprib,

Please help with a question: Can I use wildcards on the topics?

I'm refatoring a service from AkkaStrams To NodeRed, and in Java I use a wildcard * for consumer subscription:

Consumer.committableSource(consumerSettings, Subscriptions.topics("realms.*")){
         ...
}

But when I used on Consumer Group Node it not received the message: image

My debug nodes: image image

If I set the exact topic it works fine: image

I don't know kafka-node, so I don't know if wildcards works there.

peterprib commented 3 years ago

Hi, such a consideration would have to be delivered by Kafka server to work seamlessly. My understanding, it doesn't. It could be emulated by the consumer process continually polling the server and reacting to new topics added. Could add such a feature if demand was strong enough. It would require a new parameter for frequency of poll and would have to handle both topics added and deleted. Must admit not sure if it would be a robust process.

peterprib commented 3 years ago

Hi, thought about it and decided it was a worthwhile thing as kafka not likely to provide in short term. Latest version should allow willdcards using regex format. Was a bit more effort than I originally thought.

janainascal commented 3 years ago

Hey Peter, Thanks a lot!! I will test it.