New groups don't consume old messages

[Bios-Marcel]

Maybe this is a missunderstanding as to how this is supposed to work, sorry if so.

I have the following usecase:

• N messages are published to topic A (potentially by an external service)
• Service X starts listening to topic A using a fresh group ID (let's say a random UUID V4)

In this scenario, I will not receive any messages. Only if I publish new messages after having started to listen.

I've already tried to set config.Consumer.Offsets.Initial = sarama.OffsetOldest, expecting to start at 0, as it should be the oldest. Especially since no one has ever read from the topic before. And offsets should be separate for groups either way, if I understood correctly?

A workaround I found, was manually resetting the offsets to 0, however, this isn't really a good solution I think?

If you think that this sounds like a bug, I can provide you with an extensive example repository.

[github-actions[bot]]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

[dnwe]

@Bios-Marcel config.Consumer.Offsets.Initial = sarama.OffsetOldest should have been sufficient, but crucially only if the consumer hasn't already committed an offset newer than that. As long as you start a brand new consumer group with that then it should begin from the first available offset

[Bios-Marcel]

Well, since I couldn't figure out what was going on, I actually went for franz-go instead of Sarama. But thanks for the answer anyway.