Does jmx_exporter able to fetch consumer metrics?

[DesireWithin]

My kafka version is kafka_2.13-2.8.1. I'm using jmx_prometheus_javaagent-0.16.1.jar. I notice there are two kafka relevant example config yaml: kafka-2_0_0.yml and kafka-connect.yml So I put them into one yaml as my jmx_exporter config. After I restart kafka broker, It seems only kafka-2_0_0.yml take effect, I can't search any kafka_consumer* metrics. What can I do?

[dhoard]

If properly configured, the export will show Kafka consumer metrics.

Consumer lag is not part of standard Kafka consumer metrics. You will need to look at something like Kafka Lag Exporter (https://github.com/seglo/kafka-lag-exporter) for consumer lag.

[jasontran2022]

@dhoard Double checking and clarifying to see if the metrics from https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchMetricsManager.java#L72 in the form of kafka.consumer:type=consumer-fetch-manager-metrics,partition="{partition}",topic="{topic}",client-id="{client-id}" for records-lag, records-lag-max, and records-lag-avg are not supported by this jmx_exporter? I'm able to see these metrics in the metric output log, but it always report back as either 0 or NaN even though looking the Kafdrop UI or Kafka CLI shows non-zero values.

[dhoard]

@jasontran2022 MBeans with numeric attributes should be exposed. Have you inspected with JConsole to confirm the MBean attribute type/value?

[jasontran2022]

Yes, I have inspected with JConsole to confirm those MBeans attribute type. All three of them are type of double. Also, I'm using the default configuration yaml to expose all metrics and using the jar version 0.20.0

lowercaseOutputName: true
rules:
  - pattern: ".*"

Below is a screenshot of the records-lag-max MBean info:

[dhoard]

@jasontran2022 Per your screenshot, the value is shown in JConsole is NaN, which explains why the exporter isn't exposing the value. There is nothing the exporter can do with the value.

Here is a similar issue I found, https://github.com/prometheus/jmx_exporter/issues/609 which might provide some guidance.

[jasontran2022]

Thanks for the help, I've also tried the steps and configuration from that issue and the value from JConsole is still showing NaN even though Kafdrop UI or Kafka CLI shows non-zero values.