flipp-oss / deimos

Framework to work with Kafka, Avro and ActiveRecord
Other
60 stars 22 forks source link

Lag calculation can be incorrect #4

Closed dorner closed 4 years ago

dorner commented 5 years ago

Currently, Deimos's lag reporting is calculated based on the current offset and latest offset. Latest offset is only updated when a message comes in.

If no messages are being consumed (e.g. some bug in the client library or a crash), the lag calculation will not try to re-retrieve the latest offset. This means that lag is increasing but the lag reporter won't realize.

Change this so latest offset is always recalculated every heartbeat.