search

dashbitco / broadway_kafka

A Broadway connector for Kafka
222 stars 52 forks source link

Allow to force consume the topic from the beginning or the end #129

Closed danmarcab closed 10 months ago

danmarcab commented 10 months ago

The use case is always consuming the topic from beginning (to read the whole topic) or from the end (you only care about new messages), regardless of what the committed offsets were. See more info in https://github.com/dashbitco/broadway_kafka/issues/119

To support both cases we can add a config option begin_offset with the values :assigned and :reset and the following behaviour.

offset_reset_policy begin_offset outcome
:earliest :reset always read from start
:latest :reset always read from end
:earliest :assigned read from committed offset, or from start if not defined or expired (current behaviour)
:latest :assigned read from committed offset, or from end if not defined or expired (current behaviour)