Duplicate metric names when scraping metrics on Kafka clients

[mimaison]

When using the reporter with Kafka clients, you get the following exception when querying the metrics endpoint:

An Exception occurred while scraping metrics: java.lang.IllegalStateException: kafka_consumer_app_info_commit_id: duplicate metric name.
    at io.prometheus.metrics.model.registry.PrometheusRegistry.scrape(PrometheusRegistry.java:75)
    at io.prometheus.metrics.exporter.common.PrometheusScrapeHandler.scrape(PrometheusScrapeHandler.java:112)
    at io.prometheus.metrics.exporter.common.PrometheusScrapeHandler.handleRequest(PrometheusScrapeHandler.java:53)
    at io.prometheus.metrics.exporter.httpserver.MetricsHandler.handle(MetricsHandler.java:43)
    at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:95)
    at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
    at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:98)
    at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:851)
    at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:95)
    at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:818)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)

[mimaison]

To reproduce this issue, you need to start multiple instances of the same type of Kafka client (producer, consumer or admin) in a JVM. You can either:

• write a small application (be sure each instance is configured with the reporter)
• use Connect and start 2 sink or 2 source connectors
• use Streams with a stateful topology, for example the quickstart app: https://kafka.apache.org/38/documentation/streams/quickstart

[OwenCorrigan76]

@mimaison I'm missing something. I'm running the Streams example mentioned above with the wordcount application working and each instance of producer and consumer configured with the reporter. There is no issue with duplicate warnings.

I've also tried using multiple server.properties with a different port for each.

For instance: server.properties has prometheus.metrics.reporter.listener = http://localhost:8081 and server.properties1 has prometheus.metrics.reporter.listener = http://localhost:8082

Does that count as having multiple instances on the same JVM?

This is the producer command I'm running for example:

./bin/kafka-console-producer.sh --producer.config ./config/producer.properties --bootstrap-server localhost:9092 --topic streams-plaintext-input

and this is the consumer:

./bin/kafka-console-consumer.sh --consumer.config ./config/consumer.properties --bootstrap-server localhost:9092 \
    --topic streams-wordcount-output \
    --from-beginning \
    --property print.key=true \
    --property print.value=true \
    --property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer \
    --property value.deserializer=org.apache.kafka.common.serialization.LongDeserializer

This is a sample of the consumer output so the wordcount app is working:

streams 8
lead    8
to  8
kafka   8

Any advice welcome. Thanks

[OwenCorrigan76]

I've tried the other approach and created a KafkaProducer that runs two producers at the same time.

I keep getting this error in my producer log when trying to run it the metric reporter added to the configs:

Exception in thread "main" java.lang.NullPointerException: Cannot invoke "java.util.Optional.ifPresent(java.util.function.Consumer)" because "this.clientTelemetryReporter" is null

Here is my producer: https://gist.github.com/OwenCorrigan76/39c5da6f3b18dc980140149b41b68a79

I came across this when looking into it: https://issues.apache.org/jira/browse/KAFKA-17478

Any help appreciated. @k-wall @mimaison

[mimaison]

To reproduce with Streams, just run:

bin/kafka-run-class.sh org.apache.kafka.streams.examples.wordcount.WordCountDemo ./config/streams.properties

where ./config/streams.properties contains the configurations listed in https://github.com/strimzi/metrics-reporter#kafka-connect-and-kafka-streams

Then open http://localhost:8080/metrics.

Regarding the producer issue you hit, that's a bug upstream. So either try with an earlier version or newer version. You can also reproduce this issue with consumers or Connect.

[OwenCorrigan76]

@mimaison Thank you