search

redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.16k stars 559 forks source link

Doc: How to monitor consumer and producer behavior #3430

Open coral-waters opened 2 years ago

coral-waters commented 2 years ago

Need to include info on how to "monitor things like consumer lag (per topic, or per partition, time would be ideal), consume rate, and produce rate. Going through the metrics currently available on docker do not seem to provide these metrics (at least not from cmd+f for some common terms from the metrics endpoint output)."

This is related to: publish consumer group metrics via prometheus #1275

Discussion here: https://github.com/vectorizedio/redpanda/discussions/3419#discussioncomment-1942888

JIRA Link: CORE-814

gyfis commented 2 years ago

FWIW, this is how we calculate group lag:

vectorized_cluster_partition_high_watermark - on(instance, topic, partition) vectorized_kafka_group_offset