li-apache-kafka-clients is a wrapper library for the Apache Kafka vanilla clients. It provides additional features such as large message support and auditing to the Java producer and consumer in the open source Apache Kafka.
BSD 2-Clause "Simplified" License
131
stars
52
forks
source link
Added a new consumer metric - consumerOffsetWatermarkSpan to represent "distance between the safe offset and watermark" #178
In order to identify a partition "stall" due to partial large message assembly (and not due to other problems in the consumer), we are introducing a metric called "consumer offset watermark span". For each partition, the span distance is computed as (watermark minus safeOffset). Consumer offset watermark span is defined as the sum of all span distance for assigned partitions that have currently undelivered large messages in the consumer.
When the consumer is progressing with no partial large messages in buffer, its span distance should be 0 because watermark is same as safeOffset.
In order to identify a partition "stall" due to partial large message assembly (and not due to other problems in the consumer), we are introducing a metric called "consumer offset watermark span". For each partition, the span distance is computed as (
watermark minus safeOffset). Consumer offset watermark span is defined as the sum of all span distance for assigned partitions that have currently undelivered large messages in the consumer. When the consumer is progressing with no partial large messages in buffer, its span distance should be 0 because watermark is same as safeOffset.