search

exasol / kafka-connector-extension

Exasol Kafka Extension for accessing Apache Kafka
MIT License
4 stars 7 forks source link

Allow the specification of the offset reset strategy #26

Closed baunz closed 3 years ago

baunz commented 3 years ago

When no previous offset is present (because it is the first load) or the last offset gathered from the table is out of range (because the last load was longer ago than the topic retention time), the connector always starts from the end of topic (kafka default strategy 'latest'). For low throughput topics there might be no message at that point of time so again, no new offset is stored and the messages are lost. Additionally, for topics that work with compaction it is expected to load everything from beginning the topic.

This commit allows specifying AUTO_OFFSET_RESET as property and the value is passed to the kafka consumer. The default setting is changed to earliest to prevent this behaviour. When one really wants to consume from the topic end, it can be overriden.

morazow commented 3 years ago

Thank @baunz for the pull request! Going to merge now.

baunz commented 3 years ago

Hey @morazow ,

thank again for the fast response. I have one small requirement left but I will first open an issue to discuss this with you.

Cheers, Johannes