search

micronaut-projects / micronaut-kafka

Integration between Micronaut and Apache Kafka
Apache License 2.0
86 stars 107 forks source link

Programatically adding kafka consumers #321

Open ckosmowski opened 3 years ago

ckosmowski commented 3 years ago

The use of @KafkaListener annotation is very convenient if we use the micronaut-kafka api directly. However if we want so provide our own classes that offer pre-configured kafka consumers we would have to implement the poll loop ourselfs allthough all the beautiful code is already there in KafkaConsumerProcessor.

I'd like to suggest a feature where we would be able to add a preconfigured kafkaConsumer to KafkaConsumerProcessor that reuses the code that is already there in the process() method but takes the configuration that is handed as parameters instead as the annotation information.

As a more specific example we'd like to be able to have a KafkaListener where offsetStrategies and all the parameters are configured, also the KafkaListenerException handler is set properly. The User (i.E. Developers of other teams) should then be able to just specify a topic, a groupId and how they'd like to process the messages. They shouln't care for committing or resetting offsets in case of an Exception or such things. This would be a great benefit for us.

graemerocher commented 3 years ago

Sounds like a nice idea. PRs are of course welcome 🙏

ckosmowski commented 3 years ago

If "PRs" is short for prayers i'll send you some. If you mean pull requests this feature might actually be a good one to start contributing. I'll do my very best.

mcac0006 commented 3 years ago

I've been using micronaut-kafka for 2 years but this is the first time I would be contributing to the code base. Nonetheless, I would like to lend a hand.

However if we want so provide our own classes that offer pre-configured kafka consumers we would have to implement the poll loop ourselfs allthough all the beautiful code is already there in KafkaConsumerProcessor.

This would help - do you happen to have a code snippet please?

awattez commented 1 year ago

hello, I find this feature very useful in a reactive context, it is the exact definition of reactive.

Let's imagine creating consumers on the fly in order to react to a message or other... do you have any news or leads for implementing programmatic consumers that will be accessible through the ConsumerRegistry and other useful components of micronaut-kafka?

marcosflobo commented 2 months ago

Any news on this?