Can multiple consumers get same message when using Consumer Group?

[renanliberato]

Hi!

I noticed that the README states when using consumer groups we do not need to manage the offset. However, it wasn't clear if it deals fine with multiple consumers getting messages from the same topic. Do we have a chance of multiple consumers (multiple goroutines, for example) getting the same message (and we should implement this idempotent logic in the application)? Or is it handled too by kafka-go or Kafka itself?

I tried looking through the issues and in the tests, but I couldn't find an explicit answer for it. Sorry if it's some duplicated issue or not the ideal place.

[achille-roussel]

Hello @renanliberato!

It is possible for consumers to observe messages more than once, this typically happens during consumer group rebalance when partitions are reshuffled between consumers, which races with commits happening asynchronously.

The consumer guarantees are "at least once", in most cases each message will be seen once but there are edge cases where some messages may be seen more than once.

Let me know if the answer was useful!