search

danielqsj / kafka_exporter

Kafka exporter for Prometheus
Apache License 2.0
2.1k stars 602 forks source link

Add new metric for kafka topic partition size. #298

Open xxm404 opened 2 years ago

xxm404 commented 2 years ago

New metric

kafka_topic_partition_size{partition="30",topic="__consumer_offsets"} 2443
kafka_topic_partition_size{partition="31",topic="__consumer_offsets"} 1031
kafka_topic_partition_size{partition="32",topic="__consumer_offsets"} 1738
kafka_topic_partition_size{partition="33",topic="__consumer_offsets"} 0
kafka_topic_partition_size{partition="34",topic="__consumer_offsets"} 0
kafka_topic_partition_size{partition="35",topic="__consumer_offsets"} 614
kafka_topic_partition_size{partition="36",topic="__consumer_offsets"} 578
kafka_topic_partition_size{partition="37",topic="__consumer_offsets"} 24920
kafka_topic_partition_size{partition="38",topic="__consumer_offsets"} 17611
kafka_topic_partition_size{partition="39",topic="__consumer_offsets"} 9930
xxm404 commented 2 years ago

@danielqsj hi, I add a new metric for a topic partition size, like this

kafka_topic_partition_size{partition="30",topic="__consumer_offsets"} 2443
kafka_topic_partition_size{partition="31",topic="__consumer_offsets"} 1031
kafka_topic_partition_size{partition="32",topic="__consumer_offsets"} 1738
kafka_topic_partition_size{partition="33",topic="__consumer_offsets"} 0
kafka_topic_partition_size{partition="34",topic="__consumer_offsets"} 0
kafka_topic_partition_size{partition="35",topic="__consumer_offsets"} 614
kafka_topic_partition_size{partition="36",topic="__consumer_offsets"} 578
kafka_topic_partition_size{partition="37",topic="__consumer_offsets"} 24920
kafka_topic_partition_size{partition="38",topic="__consumer_offsets"} 17611
kafka_topic_partition_size{partition="39",topic="__consumer_offsets"} 9930

please review this PR

atrbgithub commented 2 years ago

This looks great 👍

atrbgithub commented 2 years ago

@xxm404 I've had a look at building on your code to make things a little faster as it was making a call for every partition, which was taking quite some time on a cluster with many topics/partitions.

Please see here

This builds a map of all the brokers and topics, with an array for each partition they own.

It then makes a single call for each broker with an array of topics and partitions which that broker is the leader for:

https://github.com/atrbgithub/kafka_exporter/blob/5923381978656a6fbeb91fb069c14f44ecb5f73e/kafka_exporter.go#L607

Not sure if you want to modify your PR to do something similar?

soloradish commented 2 years ago

It's better to follow the metric's naming convention. Add a unit as this metric's suffix like kafka_topic_partition_size_bytes.

But I couldn't managed to find the right unit name in Kafka java client docs. Maybe you guys can figure it out ^-^

@xxm404 @atrbgithub

turbotankist commented 9 months ago

Any updates?

soloradish commented 9 months ago

Any updates?

You can use my build, https://github.com/soloradish/kafka_exporter

It expose the kafka_topic_partition_size_bytes metric. but be ware I remove all consumer related metrics, because I use kafka lag exporter for these.