search

redpanda-data / kminion

KMinion is a feature-rich Prometheus exporter for Apache Kafka written in Go. It is lightweight and highly configurable so that it will meet your requirements.
MIT License
610 stars 122 forks source link

consumer group offsets by partition missing #174

Closed mdkouki closed 1 year ago

mdkouki commented 1 year ago

Hi, First of all a big Thank you for the work you are doing in this project, with kafka minion we used to have this metric "kafka_minion_group_topic_partition_offset" that was useful to see the group position in a specific topic - partition log , now kminion_kafka_consumer_group_offset_commits_total (a scope of the consumer group with all topics) and kminion_kafka_consumer_group_topic_offset_sum ( topic-consumer group scope) give I larger scope idea about the offsets but since the kafka unit scope is the partition we miss that detail: the offset in the partitions level is a valuable information I think

is there a way in kminion to get that same information (the offset reached by a consumer group in the level of partitions of one topic) ?

Thanks in advance

weeco commented 1 year ago

Yes there, is the following metric:

# HELP kminion_kafka_consumer_group_topic_partition_lag The number of messages a consumer group is lagging behind the latest offset of a partition
# TYPE kminion_kafka_consumer_group_topic_partition_lag gauge
kminion_kafka_consumer_group_topic_partition_lag{group_id="bigquery-sink",partition_id="10",topic_name="shop-activity"} 147481

see: https://github.com/redpanda-data/kminion/blob/master/docs/metrics.md . The metric you mentioned is about the number of offset commits.

mdkouki commented 1 year ago

yes indeed it could be computed like this : kminion_kafka_topic_partition_high_water_mark - kminion_kafka_consumer_group_topic_partition_lag

Thanks.