search

exasol / kafka-connector-extension

Exasol Kafka Extension for accessing Apache Kafka
MIT License
4 stars 7 forks source link

#41: Fixed bug related to consuming all offsets from stale topic #42

Closed jwarlander closed 3 years ago

jwarlander commented 3 years ago

When poll returns an empty result, fetch the current partition offset from Kafka instead of falling back to our own internal "latest" offset that we read from the topic at some point in the past.

This works around the case where, if we have a topic that used to have some records, but they were all expired, the consumer would keep looping infinitely.

It happens this way since the start and end offset of our empty-but-once-populated partition will be the same, 10 for example, but our internal latest offset is perhaps 5; and if no new records appear while we poll, we'll keep the '5', comparing it to the end offset of 10, forever assuming we need to read some more records.

morazow commented 3 years ago

Hey @jwarlander,

Thanks for the pull request!

I am going to review it soon.