confluentinc / confluent-kafka-go

Confluent's Apache Kafka Golang client
Apache License 2.0
4.63k stars 658 forks source link

Unable to create multiple consumers for different topics in the same process? #1313

Open mango009 opened 2 weeks ago

mango009 commented 2 weeks ago

I create multiple consumers running in different Go routines. These consumers consume messages on different topics.

But I notice that when one consumer is blocked processing messages, the other consumer would get timeout error on the ReadMessage function.

Is it not a practice to create multiple consumers in the same process?

OneCricketeer commented 1 week ago

Does each routine run one consumer group?

One consumer can already subscribe to a regex pattern of topic names.

Sub-processing is an alternative solution than coroutines

mango009 commented 1 week ago

Yes. Each consumer group run in it's own go routine. I know alternative is to subscribe multiple topic in one go routine. But the API doesn't preclude subscribing one topic in separate go routine. And if this is a bad practice it would helpful to state this in the document (and maybe give a reason).

OneCricketeer commented 1 week ago

Can you please share an example? Also, do they all share a config? Specifically all a single group.id