KMinion is a feature-rich Prometheus exporter for Apache Kafka written in Go. It is lightweight and highly configurable so that it will meet your requirements.
MIT License
620
stars
122
forks
source link
Add fine control on what consumer groups metrics are exposed to reduce scraped metrics in large clusters #108
For large clusters with tens of thousands of consumer groups, the number of metrics exposed may be quite large.
This in turn poses extra pressure on scraping Prometheus as all these series names need to kept in RAM.
Few suggestions:
add flag to disable exposing metrics for inactive consumer groups (Empty or Dead)
List offsets for Stable consumer groups only : reduce the number of metrics for clusters where lots are consumer groups are short-lived and empty.
Report consumer_group_empty_members metrics for stable groups only
For large clusters with tens of thousands of consumer groups, the number of metrics exposed may be quite large. This in turn poses extra pressure on scraping Prometheus as all these series names need to kept in RAM.
Few suggestions:
See https://github.com/cloudhut/kminion/pull/105#discussion_r680732016