I have a consumer and even if message processing fails I call MarkOffset but I do not commit.
Expected Behavior
The consumer would not commit any offset and the lag should be greater than or equal to 1. The current instance should continue processing next messages but if I restart the consumer it should process from the offset that was not committed.
And this is according to the documentation
// MarkOffset marks the provided offset, alongside a metadata string
// that represents the state of the partition consumer at that point in time. The
// metadata string can be used by another consumer to restore that state, so it
// can resume consumption.
//
// To follow upstream conventions, you are expected to mark the offset of the
// next message to read, not the last message read. Thus, when calling `MarkOffset`
// you should typically add one to the offset of the last consumed message.
//
// Note: calling MarkOffset does not necessarily commit the offset to the backend
// store immediately for efficiency reasons, and it may never be committed if
// your application crashes. This means that you may end up processing the same
// message twice, and your processing should ideally be idempotent.
Current Behavior
Even though Consumer.Offsets.AutoCommit.Enable is set to false and there is no Commit() call in the code. The offset is committed regardless.
Before calling MarkOffset
After calling MarkOffset without committing and no auto-commit
That does not seem like expected functionality. Or am I missing something?
Versions
Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.
Configuration
What configuration values are you using for Sarama and Kafka?
Problem Description
I have a consumer and even if message processing fails I call
MarkOffset
but I do not commit.Expected Behavior The consumer would not commit any offset and the lag should be greater than or equal to 1. The current instance should continue processing next messages but if I restart the consumer it should process from the offset that was not committed.
And this is according to the documentation
Current Behavior Even though
Consumer.Offsets.AutoCommit.Enable
is set tofalse
and there is noCommit()
call in the code. The offset is committed regardless.Before calling
MarkOffset
After calling
MarkOffset
without committing and no auto-commitThat does not seem like expected functionality. Or am I missing something?