This PR adds lag reporting to kafka-delta-ingest via statsd metric emissions.
Two flavors are added - Buffer lag and write lag. Buffer lag represents the last offsets stored in buffer. Write lag represents the last offsets written to the target delta lake table. We report four metrics for each:
total lag (sum of lag across all partitions handled by the the task/process)
max (the highest lag observed across all handled partitions)
min (like max but min)
number of partitions handled by the task
Buffer lag reports roughly every minute (but requires a message receive to trigger).
Write lag reports every time a write is performed.
This PR adds lag reporting to kafka-delta-ingest via statsd metric emissions.
Two flavors are added - Buffer lag and write lag. Buffer lag represents the last offsets stored in buffer. Write lag represents the last offsets written to the target delta lake table. We report four metrics for each:
Buffer lag reports roughly every minute (but requires a message receive to trigger). Write lag reports every time a write is performed.