FATAL ERROR in native method: processing of -javaagent failed

[Lucky38i]

I'm having issues deploying our sink connector with the JMX Exporter agent in a kubernetes environment. Regardless of what config I use, whether my own, the example config or just a very stripped down config. I continuously get the same error

Exception in thread "main" java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)
Caused by: java.lang.NoSuchFieldError: UNKNOWN
    at io.prometheus.jmx.JmxCollector$Rule.<init>(JmxCollector.java:59)
    at io.prometheus.jmx.JmxCollector.loadConfig(JmxCollector.java:216)
    at io.prometheus.jmx.JmxCollector.<init>(JmxCollector.java:93)
    at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:29)
    ... 6 more
*** java.lang.instrument ASSERTION FAILED ***: "result" with message agent load/premain call failed at src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422
FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed

The current config I'm using is jms_exporter.yaml

---
rules:
  - pattern: ".*"

My dockerfile is as follow

## SETUP ##
FROM maven:3.8.6-openjdk-18-slim as maven_dependencies

WORKDIR /app
COPY pom.xml .
RUN --mount=type=cache,target=/root/.m2 mvn dependency:copy-dependencies -DoutputDirectory=libs -Dmaven.test.skip

## RUN ##
FROM confluentinc/cp-kafka-connect:7.2.1

ARG ENVIRONMENT
ARG URL_ENV
ARG CONNECT_GROUP_ID

ENV env=$ENVIRONMENT
ENV url_env=$URL_ENV
ENV CONNECT_GROUP_ID = CONNECT_GROUP_ID
ENV KAFKA_OPTS="-javaagent:/usr/share/java/cp-base-new/jmx_prometheus_javaagent-0.18.0.jar=5556:/opt/prometheus/jmx_exporter.yaml"

# Copy Maven dependencies
COPY --from=maven_dependencies /app/libs /usr/share/java/cp-base-new/

# Copy JMX Configs
COPY jmx_exporter.yaml /opt/prometheus/jmx_exporter.yaml

# Copy init script
COPY ./init.sh init.sh

# Install Mongo Connect
RUN confluent-hub install --no-prompt mongodb/kafka-connect-mongodb:1.7.0

USER root
RUN chown appuser init.sh
RUN chown appuser /opt/prometheus
RUN chmod +x init.sh

USER appuser
CMD ["sh", "-c", "./init.sh $env $CONNECT_GROUP_ID"]

The shell script just starts the kafka connector and pushes configs to the connector.

[dhoard]

@Lucky38i This appears to be some type of mismatch between jars in the image.

The first step would be to run the container without installing the connector to rule in/out the connector.

---

Some initial code analysis ...

Caused by: java.lang.NoSuchFieldError: UNKNOWN
    at io.prometheus.jmx.JmxCollector$Rule.<init>(JmxCollector.java:59)

Is caused when code is referencing a class field or enum and it doesn't exist in the referenced class.

    static class Rule {
      Pattern pattern;
      String name;
      String value;
      Double valueFactor = 1.0;
      String help;
      boolean attrNameSnakeCase;
      boolean cache = false;
      Type type = Type.UNKNOWN; // <- reference to Type.UNKNOWN
      ArrayList<String> labelNames;
      ArrayList<String> labelValues;
    }

The Type enum is defined in the Collector class from the client_java project dependency ...

  public enum Type {
    UNKNOWN, // This is untyped in Prometheus text format.
    COUNTER,
    GAUGE,
    STATE_SET,
    INFO,
    HISTOGRAM,
    GAUGE_HISTOGRAM,
    SUMMARY,
  }

https://github.com/prometheus/jmx_exporter/blob/main/collector/src/main/java/io/prometheus/jmx/JmxCollector.java

---

Type.UNKNOWN was added to client_java version 0.16.0. If there are any jars that use older versions of the code or an older version of the jar is in the classpath, this error could occur.

[dhoard]

@Lucky38i I believe I see the issue.

ENV KAFKA_OPTS="-javaagent:/usr/share/java/cp-base-new/jmx_prometheus_javaagent-0.18.0.jar=5556:/opt/prometheus/jmx_exporter.yaml"

... is adding jmx_prometheus_javaagent-0.18.0.jar to the Connect classpath, which is incorrect.

Please test putting the exporter jar in a different directory not in the Connect classpath.

[Marchosiax]

@Lucky38i I believe I see the issue.

ENV KAFKA_OPTS="-javaagent:/usr/share/java/cp-base-new/jmx_prometheus_javaagent-0.18.0.jar=5556:/opt/prometheus/jmx_exporter.yaml"

... is adding jmx_prometheus_javaagent-0.18.0.jar to the Connect classpath, which is incorrect.

Please test putting the exporter jar in a different directory not in the Connect classpath.

I have this exact issue. I've put the jar file in a different directory yet the issue exists.

KAFKA_OPTS=-javaagent:/opt/prometheus/jmx-exporter-0.20.0.jar=1234:/opt/prometheus/kafka-jmx-exporter.yml

[dhoard]

@Marchosiax can you publish a Docker image I can use for testing?

[Marchosiax]

@dhoard It's a standard config I put the config files here so you can run locally

docker-compose.yml

version: '3.8'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.1.1
    hostname: zookeeper
    volumes:
      - zookeeper-data:/var/lib/zookeeper/data
      - zookeeper-log:/var/lib/zookeeper/log
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
      - ZOOKEEPER_CLIENT_PORT=2181
    networks:
      - default
    deploy:
      restart_policy:
        condition: on-failure
  kafka-1:
    build: docker-images/kafka
    hostname: kafka-1
    volumes:
      - kafka-1:/var/lib/kafka/data
    environment:
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
      - KAFKA_LISTENERS=CLIENT://kafka-1:29092,EXTERNAL://kafka-1:9092
      - KAFKA_ADVERTISED_LISTENERS=CLIENT://kafka-1:29092,EXTERNAL://kafka-1:9092
      - KAFKA_INTER_BROKER_LISTENER_NAME=CLIENT
      - KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE=false
      - KAFKA_OPTS=-javaagent:/opt/prometheus/jmx-exporter-0.20.0.jar=1234:/opt/prometheus/kafka-jmx-exporter.yml
    depends_on:
      - zookeeper
    networks:
      - default
    deploy:
      restart_policy:
        condition: on-failure
  kafka-2:...
  kafka-3:...

./docker-images/kafka/Dockerfile exporter jar file and and exporter config file are in this folder

FROM confluentinc/cp-kafka:7.1.1
USER root
RUN mkdir /opt/prometheus
RUN chmod +rx /opt/prometheus
COPY jmx-exporter-0.20.0.jar /opt/prometheus
COPY kafka-jmx-exporter.yml /opt/prometheus

[dhoard]

@Marchosiax while you are adding the new JMX exporter, you haven't removed the version (/usr/share/java/cp-base-new/jmx_prometheus_javaagent-0.14.0.jar) shipped in the base image

I believe /usr/share/java/cp-base-new/ is in the standard classpath, which will causes issues.

I would try to remove the version shipped in the base image (/usr/share/java/cp-base-new/jmx_prometheus_javaagent-0.14.0.jar)

[Rediska47]

Try this one. It worked for me:

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
    hostname: kafka
    volumes:
      - /home/kafka/jmx-exporter/:/usr/share/jmx-exporter
      - /home/kafka/jmx-exporter/jmx_prometheus_javaagent-0.12.0.jar:/usr/share/java/cp-base-new/jmx_prometheus_javaagent-0.12.0.jar
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
      KAFKA_JMX_OPTS: -javaagent:/usr/share/jmx-exporter/jmx_prometheus_javaagent-0.12.0.jar=1234:/usr/share/jmx-exporter/kafka-broker.yml -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false```