Deploying Latest prometheus exporter 0.17.2 getting error when starting broker

[myrulezzz]

Hi deploying prometheus exporter 0.17.2 with latest kafka yaml file getting error when starting broker. Broker is kafka packed by bitnami 3.3

Exception in thread "main" java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513) at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525) Caused by: mapping values are not allowed here in 'reader', line 27, column 66: ... le" content="{"groups": [], "environmentKey" ... ^

at io.prometheus.jmx.shaded.org.yaml.snakeyaml.scanner.ScannerImpl.fetchValue(ScannerImpl.java:920)
at io.prometheus.jmx.shaded.org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:409)
at io.prometheus.jmx.shaded.org.yaml.snakeyaml.scanner.ScannerImpl.peekToken(ScannerImpl.java:279)
at io.prometheus.jmx.shaded.org.yaml.snakeyaml.parser.ParserImpl$ParseDocumentEnd.produce(ParserImpl.java:289)
at io.prometheus.jmx.shaded.org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:171)
at io.prometheus.jmx.shaded.org.yaml.snakeyaml.comments.CommentEventsCollector$1.peek(CommentEventsCollector.java:57)
at io.prometheus.jmx.shaded.org.yaml.snakeyaml.comments.CommentEventsCollector$1.peek(CommentEventsCollector.java:43)
at io.prometheus.jmx.shaded.org.yaml.snakeyaml.comments.CommentEventsCollector.collectEvents(CommentEventsCollector.java:136)
at io.prometheus.jmx.shaded.org.yaml.snakeyaml.comments.CommentEventsCollector.collectEvents(CommentEventsCollector.java:116)
at io.prometheus.jmx.shaded.org.yaml.snakeyaml.composer.Composer.composeScalarNode(Composer.java:221)
at io.prometheus.jmx.shaded.org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:190)
at io.prometheus.jmx.shaded.org.yaml.snakeyaml.composer.Composer.getNode(Composer.java:116)
at io.prometheus.jmx.shaded.org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:142)
at io.prometheus.jmx.shaded.org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:155)
at io.prometheus.jmx.shaded.org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:477)
at io.prometheus.jmx.shaded.org.yaml.snakeyaml.Yaml.load(Yaml.java:430)
at io.prometheus.jmx.JmxCollector.<init>(JmxCollector.java:91)
at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:29)
... 6 more

java.lang.instrument ASSERTION FAILED : "result" with message agent load/premain call failed at ./src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422 FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed #

A fatal error has been detected by the Java Runtime Environment:

#

SIGSEGV (0xb) at pc=0x00007f0343491602, pid=1, tid=994

#

JRE version: OpenJDK Runtime Environment (11.0.18+10) (build 11.0.18+10-LTS)

Java VM: OpenJDK 64-Bit Server VM (11.0.18+10-LTS, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)

Problematic frame:

C [libc.so.6+0x22602] abort+0x1ee

#

Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to //core.1)

#

An error report file with more information is saved as:

/tmp/hs_err_pid1.log

#

If you would like to submit a bug report, please visit:

https://bell-sw.com/support

# Any idea?

[fstab]

This Exception is coming from the code that reads the config file. Could you post your config here?

[myrulezzz]

Hi @fstab Here is my docker-compose broker1: container_name: broker1 image: 'bitnami/kafka:latest' hostname: broker1 depends_on:

• zookeeper ports:
  • 9092:9092
  • 9093:9093
  • 1092:1092
  • 7072:7072
  • 4000:4000 environment:
  • KAFKA_BROKER_ID=1
  • KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=INTERNAL:SSL,CLIENT:SASL_SSL,UI:SSL, AIRBYTE:SASL_PLAINTEXT
  • KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
  • KAFKA_CFG_LISTENERS=CLIENT://:9092, INTERNAL://:9093,UI://:1092,AIRBYTE://:7072
  • KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://172.18.16.148:9092,INTERNAL://broker1:9093,UI://172.18.16.148:1092,AIRBYTE://172.18.16.148:7072
  • KAFKA_ZOOKEEPER_USER=kafka
  • KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INTERNAL
  • KAFKA_ZOOKEEPER_PASSWORD=kafkarac123
  • KAFKA_CLIENT_USERS=kafkauser
  • KAFKA_ZOOKEEPER_PROTOCOL=SASL
  • KAFKA_CLIENT_PASSWORDS=kafkadev123
  • KAFKA_CERTIFICATE_PASSWORD=kafkadev123
  • KAFKA_TLS_TYPE=JKS # or PEM
  • KAFKA_CONFLUENT_SCHEMA_REGISTRY_URL= http://schemaregistry1:8081
  • KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR= 3
  • KAFKA_DEFAULT_REPLICATION_FACTOR= 3
  • KAFKA_NUM_PARTITIONS= 3
  • KAFKA_LOG_RETENTION_HOURS= 96
  • KAFKA_CFG_AUTHORIZER_CLASS_NAME=kafka.security.authorizer.AclAuthorizer
  • KAFKA_CFG_ALLOW_EVERYONE_IF_NO_ACL_FOUND=false
  • EXTRA_ARGS=-javaagent:/opt/bitnami/kafka/config/certs/jmx_prometheus_javaagent-0.17.2.jar=4000:/opt/bitnami/kafka/config/certs/kafka-2_0_0.yml
  • KAFKA_CFG_SUPER_USERS=User:kafkauser;User:CN=broker1,OU=broker1,O=broker1,L=broker1,ST=broker1,C=CY;User:CN=broker2,OU=broker2,O=broker2,L=broker2,ST=broker2,C=CY;User:CN=broker3,OU=broker3,O=broker3,L=broker3,ST=broker3,C=CY;User:CN=chemaregistry1,OU=chemaregistry1,O=chemaregistry1,L=chemaregistry1,ST=schemaregistry1,C=CY volumes:
  • /mnt/datafiles/kafka/broker1ssl/:/opt/bitnami/kafka/config/certs/
  • /mnt/datafiles/kafka/broker1data:/bitnami/kafka/data Check the EXTRA_ARGS

[myrulezzz]

I have added the files in my cert folder to make sure that the files are available in the container correctly

[fstab]

I think there's something wrong in kafka-2_0_0.yml in your certs/ folder. Please post the content of this file.

[myrulezzz]

Hi @fstab. The file is this one. https://github.com/prometheus/jmx_exporter/blob/main/example_configs/kafka-2_0_0.yml

[myrulezzz]

Hi @fstab. Thanks for the help. The problem was my file. I was using another one and i realized now. I replaced with the one i send you the link and i see the metrics on port 4000 So i should be ok