kafka.server.FetcherLagMetrics are not reported

[horkhe]

In our server.properties file we specified:

external.kafka.statsd.metrics.exclude_regex=(kafka\.consumer\.FetchRequestAndResponseMetrics.*)|(.*ReplicaFetcherThread.*)|(kafka\.log\.Log\..*)|(kafka\.cluster\.Partition\..*)

But the kafka.server.FetcherLagMetrics.* are still not reported. Any insight on that will be greatly appreciate. We are using Kafka 2.10-0.8.2.0, and version 0.4.0 of the plugin.

[alexism]

Hi @horkhe

as of 0.8.2, there are 3 metrics related to lag:

• kafka.server.ReplicaFetcherManager.Replica_MaxLag
• kafka.consumer.ConsumerFetcherManager.MaxLag
• kafka.server.FetcherLagMetrics.ConsumerLag

So, indeed, kafka.server.FetcherLagMetrics.* is still a valid metric.

To investigate your issue, a few things come to my mind:

• the metrics are lazily added to the metrics registry. So you need a test case that triggers the code path to initialize the metrics. I know it sounds obvious but I got caught by that in the past. In your case, you would need an active consumer. See https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L138 and https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L256
• to debug your issue, we could first check if the metric is actually reported with tcpdump -As 0 -i lo port 8125 and udp|grep FetcherLagMetrics. JConsole would work to.
• I would eventually enable more logging for the ExcludeMetricPredicate. This would be something like come.airbnb.metrics.ExcludeMetricPredicate.level=trace. See https://github.com/airbnb/kafka-statsd-metrics2/blob/master/src/main/java/com/airbnb/metrics/ExcludeMetricPredicate.java#L46 . This will be quite verbose, so do it in dev.
• if the logs show that FetcherLagMetrics is indeed excluded, then the regex must be double checked, or there's a bug.
• if FetcherLagMetrics does not show up in the logs, and there is an active consumer, I'd double check tcpdump and JConsole.

I hope it helps,

Alexis

[horkhe]

@alexism man, I should have run tcpdump before creating this issue. I do see FetcherLagMetrics traveling to our statsd server. I still need to understand why this metric does not appear in the Grafana UI when other do, but that is not a problem of the plugin.

Thank you very much for the detailed answer and the great plugin.