IBM / sarama

Sarama is a Go library for Apache Kafka.
MIT License
11.21k stars 1.73k forks source link

How can consumer group consume new partition without losing data using OffsetNewest config? #2927

Open AlexanderJLiu opened 3 weeks ago

AlexanderJLiu commented 3 weeks ago

sarama config:

config.Consumer.Offsets.Initial = sarama.OffsetNewest

Consumer group consume topic partitions using OffsetNewest config, when topic partitions increase, producers have produced data to new partitions, but consumer group haven't rebalance to Stable state, when consumer ready, consumer would consume new partitions begining with newest data, data produced to new partitions between rebalance would lose.

Is there any config to control consume newer partitions with OffsetOldest config while other partitions with OffsetNewest config?