search

omarsmak / kafka-consumer-lag-monitoring

Client tool that exports the consumer lag of Kafka consumer groups to Prometheus or your terminal
MIT License
50 stars 13 forks source link

Question : metrics equivalence with AWS MSK consumer lag metrics #34

Closed raphaelauv closed 2 years ago

raphaelauv commented 2 years ago

Is your feature request related to a problem? Please describe. Hello , thanks for your tooling , we are currently using aws MSK, that provide natively metrics for the consumer lags

https://docs.aws.amazon.com/msk/latest/developerguide/consumer-lag.html

EstimatedMaxTimeLag, EstimatedTimeLag, MaxOffsetLag, OffsetLag, and SumOffsetLag

time based lag metrics and "size" based lag metrics

Since we would like to move to our own kafka, we would like a tool to get theses metrics. And looking to your project :+1:

Describe the solution you'd like Could you detail if your metrics are equivalent to the metrics that we are currently using with aws msk

Describe alternatives you've considered Before the release of that feature inside aws MSK, we were using burrow ( in 2020 ) and it was less precise and complete

omarsmak commented 2 years ago

Contributions are welcome to add this feature :)

raphaelauv commented 2 years ago

Hey , it's not a feature request , I wanted to know what kind of consumer-group lag metrics compute your project. Thanks

omarsmak commented 2 years ago

Sorry @raphaelauv misread your request. Regarding your question, this tool provides only the lag offset (number of messages) either per partition or total lag offset per to topic, which makes it simple yet helpful since the offset lag is an important metrics to monitor Kafka topics. However, it doesn't give a time lag similar to MSK metrics. In the readme, there is an overview on what is exposed through this tool. You can give it a try and see if it can suit your requirements :).