In KafkaConsumerConsumeNum, we forget errors when we've already seen some messages,
because we can only return an error or messages, and the messages are more important.
Additionally, the error could be re-fetched when fetching another time at the higher offset, as no
new messages will be available then and the error will thus be returned.
This logic breaks when returning EOFs: EOFs will be returned even if we're at the end of a partition.
But it's still important to see errors, and arguably it's more important.
So, if we've only seen EOFs so far, we should return the error when encountering one.
This would help us in receiving OffsetOutOfRange errors, which is important for us to detect.
iradul
commented
2 years ago
In KafkaConsumerConsumeNum, we forget errors when we've already seen some messages, because we can only return an error or messages, and the messages are more important.
Additionally, the error could be re-fetched when fetching another time at the higher offset, as no new messages will be available then and the error will thus be returned.
This logic breaks when returning EOFs: EOFs will be returned even if we're at the end of a partition. But it's still important to see errors, and arguably it's more important. So, if we've only seen EOFs so far, we should return the error when encountering one.
This would help us in receiving OffsetOutOfRange errors, which is important for us to detect.