search

softwaremill / kmq

Kafka-based message queue
https://softwaremill.com/open-source/
Apache License 2.0
330 stars 46 forks source link

EndOffset of a redelivery consumer #12

Open shubhamDhoble opened 6 years ago

shubhamDhoble commented 6 years ago

Just wanted to know the use of the disableRedeliveryBefore argument of the marker queue?

And as per the logic, I feel the following case scenario would be a failure. for code reference : https://github.com/softwaremill/kmq/blob/391f9f9e72c09d8cd0b44bab0b5c9d776389c8dd/core/src/main/scala/com.softwaremill.kmq/redelivery/ConsumeMarkersActor.scala#L71

Case Scenario: For a Topic(X) there is a marker topic(MT). One redelivery tracker is listening to the marker topic. The order of markers on a partition are SM1, SM2, EM1, EM2, where SMx is start-marker for some key x and EMx is end-marker for some key x. the redelivery tracker commits on Kafka till EM2. Then redelivery tracker disconnects and a new redelivery consumer is created, and in between a new SM3 is published. But as the consumer connects after SM3, endOffset/ disabledRedeliveryBefore for MarkerQueue of partition is after the SM3 offset.

Problem: SM3 would not be tracked by any redelivery tracker.

Possible Solution: Rather than endOffset, the redelivery tracker can disable redelivery before the last committed offset.

adamw commented 6 years ago

The disableRedeliveryBefore argument is there to enable redeliveries only after all known data has been loaded from Kafka to build the MarkersQueue state on startup or after a restart.

It doesn't affect which markers are tracked or not. So in your example, SM3 would still be tracked and considered for redelivery (provided that there's some movement on the topic, and more markers are seen).

The setting only affects when markers will be checked for redeliveries.

shubhamDhoble commented 6 years ago

Ok, I agree SM3 would be tracked but it would not be redelivered as redelivery would be disabled for that offset. This should not be the required behavior as redelivery is required to be done, as it was not even done by the first instance of redelivery tracker.

adamw commented 6 years ago

It would be redelivered, but possibly not immediately, but at a later time, after some more markers are published to the topic.