search

prometheus / jmx_exporter

A process for exposing JMX Beans via HTTP for Prometheus consumption
Apache License 2.0
3.06k stars 1.2k forks source link

kafka_2.13-3.7.1 without metrics #984

Closed qq359130530 closed 3 months ago

qq359130530 commented 4 months ago

``OS: Linux JDK：1.8.0_341 Kafka：kafka_2.13-3.7.1 jmx_exporter：1.0.1

edit kafka-server-start.sh image

start Kafka, Kafka Server normal JMX port normal image

But, metrics not running. image

dhoard commented 4 months ago

Can you provide the output of netstat -an | grep LISTEN | grep -v LISTENING?

qq359130530 commented 4 months ago

image

dhoard commented 4 months ago

The command looks incorrect.

Please provide formatted text format in the future. Images are not searchable, etc.

qq359130530 commented 3 months ago

I tried to run the Kafka & JMX Exporter console under Windows using IDEA to output the following error message:

Failed to start Prometheus JMX Exporter

java.lang.IllegalStateException: Can't register jmx_exporter_build because a metric with that name is already registered. at io.prometheus.metrics.model.registry.PrometheusRegistry.register(PrometheusRegistry.java:26) at io.prometheus.metrics.core.metrics.Metric$Builder.register(Metric.java:60) at io.prometheus.metrics.core.metrics.MetricWithFixedMetadata$Builder.register(MetricWithFixedMetadata.java:47) at io.prometheus.jmx.BuildInfoMetrics.register(BuildInfoMetrics.java:60) at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:53) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513) at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)

Prometheus JMX Exporter exiting

image

dhoard commented 3 months ago

@qq359130530

For Linux

did you change the configuration on Linux as I described?

For Windows

Do you have the full command line when running in IntelliJ on Windows?

I don't use Windows, but I suspect this is a configuration issue as well... most likely the exporter Java agent is being loaded twice.

qq359130530 commented 3 months ago

I originally wanted to run Kafka & JMX Exporter Agent on Linux, but Kafka started without exposing the /metrics service, so I tried debugging it on Windows only to see this error message with the following command-line arguments：

-XX:G1ConcRefinementThreads=13 -XX:GCDrainStackTargetSize=64 -XX:InitialHeapSize=263742592 -XX:MaxHeapSize=4219881472 -XX:+PrintCommandLineFlags -XX:ReservedCodeCacheSize=251658240 -XX:+SegmentedCodeCache -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:-UseLargePagesIndividualAllocation

dhoard commented 3 months ago

Can you provide the full command line that's being executed?

java.exe ...

qq359130530 commented 3 months ago

this is Linux cmd:

/home/worker/java/jdk/bin/java -Xmx1G -Xms1G -javaagent:/home/worker/kafka/jmx_prometheus_javaagent-1.0.1.jar=12345:/home/worker/kafka/config.yaml -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt.headless=true -Xloggc:/home/worker/kafka/kafka-2.12-2.0.1/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/home/worker/kafka/kafka-2.12-2.0.1/bin/../logs -Dlog4j.configuration=file:/home/worker/kafka/kafka-2.12-2.0.1/bin/../config/log4j.properties -cp /home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/activation-1.1.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/aopalliance-repackaged-2.6.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/argparse4j-0.7.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/audience-annotations-0.12.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/caffeine-2.9.3.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/checker-qual-3.19.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/commons-beanutils-1.9.4.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/commons-cli-1.4.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/commons-collections-3.2.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/commons-digester-2.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/commons-io-2.11.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/commons-lang3-3.12.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/commons-logging-1.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/commons-validator-1.7.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/connect-api-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/connect-basic-auth-extension-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/connect-json-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/connect-mirror-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/connect-mirror-client-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/connect-runtime-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/connect-transforms-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/error_prone_annotations-2.10.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/hk2-api-2.6.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/hk2-locator-2.6.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/hk2-utils-2.6.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-annotations-2.16.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-core-2.16.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-databind-2.16.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-dataformat-csv-2.16.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-datatype-jdk8-2.16.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-jaxrs-base-2.16.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-jaxrs-json-provider-2.16.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-module-afterburner-2.16.2.jar/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-module-jaxb-annotations-2.16.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-module-scala_2.13-2.16.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jakarta.activation-api-1.2.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jakarta.annotation-api-1.3.5.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jakarta.inject-2.6.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jakarta.validation-api-2.0.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jakarta.xml.bind-api-2.3.3.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/javassist-3.29.2-GA.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/javax.activation-api-1.2.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/javax.annotation-api-1.3.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/javax.servlet-api-3.1.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/javax.ws.rs-api-2.1.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jaxb-api-2.3.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jersey-client-2.39.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jersey-common-2.39.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jersey-container-servlet-2.39.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jersey-container-servlet-core-2.39.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jersey-hk2-2.39.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jersey-server-2.39.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-client-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-continuation-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-http-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-io-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-security-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-server-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-servlet-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-servlets-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-util-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-util-ajax-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jline-3.25.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jopt-simple-5.0.4.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jose4j-0.9.4.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jsr305-3.0.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka_2.13-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-clients-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-group-coordinator-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-group-coordinator-api-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-log4j-appender-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-metadata-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-raft-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-server-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-server-common-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-shell-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-storage-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-storage-api-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-streams-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-streams-examples-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-streams-scala_2.13-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-streams-test-utils-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-tools-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-tools-api-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-transaction-coordinator-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/lz4-java-1.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/maven-artifact-3.9.6.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/metrics-core-2.2.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/metrics-core-4.1.12.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/netty-buffer-4.1.110.Final.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/netty-codec-4.1.110.Final.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/netty-common-4.1.110.Final.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/netty-handler-4.1.110.Final.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/netty-resolver-4.1.110.Final.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/netty-transport-4.1.110.Final.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/netty-transport-classes-epoll-4.1.110.Final.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/netty-transport-native-epoll-4.1.110.Final.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/netty-transport-native-unix-common-4.1.110.Final.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/opentelemetry-proto-1.0.0-alpha.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/osgi-resource-locator-1.0.3.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/paranamer-2.8.jar:/home/worker/kafkakafka-2.12-2.0.1/bin/../libs/pcollections-4.0.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/plexus-utils-3.5.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/protobuf-java-3.23.4.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/reflections-0.10.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/reload4j-1.2.25.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/rocksdbjni-7.9.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/scala-collection-compat_2.13-2.10.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/scala-java8-compat_2.13-1.0.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/scala-library-2.13.14.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/scala-logging_2.13-3.9.4.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/scala-reflect-2.13.14.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/slf4j-api-1.7.36.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/slf4j-reload4j-1.7.36.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/snappy-java-1.1.10.5.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/swagger-annotations-2.2.8.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/trogdor-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/zookeeper-3.8.4.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/zookeeper-jute-3.8.4.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/zstd-jni-1.5.6-3.jar -Djava.security.krb5.conf=/home/worker/krb5/krb5/var/krb5kdc/krb5.conf -Djava.security.auth.login.config=/home/worker/kafka/kafka/config/kafka_client_jaas.conf kafka.Kafka /home/worker/kafkakafka-2.12-2.0.1/config/server.properties

qq359130530 commented 3 months ago

JMX Exporter require JDK 11

dhoard commented 3 months ago

JMX Exporter requires JDK 11

Java 8 is supported and tested.

Here is the list of Java containers tested:

https://github.com/prometheus/jmx_exporter/blob/main/integration_test_suite/integration_tests/src/test/resources/java-docker-images.txt