getsentry / arroyo

A library to build streaming applications that consume from and produce to Kafka.
https://getsentry.github.io/arroyo/
Apache License 2.0
41 stars 7 forks source link

feat: Enable automatic latency recording for consumers #308

Closed lynnagara closed 10 months ago

lynnagara commented 10 months ago

The goal of this change is to record a common latency metric for all consumers that use Arroyo.

The new metric is called arroyo.consumer.latency -- it represents the time in seconds from when the message was initially produced (broker timestamp) to when the consumer successfully committed the offset for a message.

Note that if a consumer is doing either manual batching (not using Arroyo's built in Reduce/Batch) or manual committing of offsets (not using Arroyo's CommitOffsets strategy), then the metric will not be recorded automatically.

Closes https://github.com/getsentry/arroyo/issues/307