search

IBM / sarama

Sarama is a Go library for Apache Kafka.
MIT License
11.41k stars 1.75k forks source link

Using Same Group ID with Multiple Consumer Groups Causes Problems #2975

Open ugrkm opened 2 weeks ago

ugrkm commented 2 weeks ago
Description

I use ConsumerGroups to consume kafka messages. I have two instances that run on kubernetes. I give the same group ID to both instances. This is to have fault tolerance (one of the pod goes down, the other should be able to keep consuming). What happens is, two instances run side by side. One of them consumes messages alright. But, when I kill one of the pods, the other one can't keep consuming. My guess is maybe broker only accepts one of the pod's ConsumerGroup and just ignores the other one? Could someone please clarify the behavior?

Versions
Sarama Kafka Go
v1.43.2 3.5.1 1.22.2
Configuration
config.Consumer.Offsets.AutoCommit.Enable = false
config.Consumer.Offsets.Initial = sarama.OffsetOldest
config.Consumer.Return.Errors = true
lbydev commented 5 days ago

Does your current topic have only one partition? If so, only one consumer will be able to consume messages. Could you check the number of partitions in your topic?

puellanivis commented 5 days ago

Does your current topic have only one partition? If so, only one consumer will be able to consume messages. Could you check the number of partitions in your topic?

I mean, even if this were the case, it should redistribute partitions to the consumers if one of them fails, right?

lbydev commented 5 days ago

Thank you for your reply. I apologize for the misunderstanding. You are correct that Kafka should redistribute partitions among the remaining consumers in the group when one of them fails.