JMX metrics not working on cluster with Jolokia JVM agent attached

[dseravalli]

All my JMX metrics are 0's on a new cluster I've added to CMAK monitoring. I have other clusters in CMAK with working metrics. I think the only difference is the new cluster has Jolokia attached to the process. CMAK v3.0.0.4 and Kafka v2.4.1

Telnet to JMX port from CMAK host works:

-> telnet broker-1 1099
Trying 172.xx.xx.xx...
Connected to broker-1.
Escape character is '^]'.

Kafka broker systemd config file:

[Unit]
Description=Kafka

[Service]
Type=simple
WorkingDirectory=/opt/kafka
User=kafka
Group=kafka
Environment=KAFKA_HEAP_OPTS="-Xmx5G -Xms5G"
Environment=JMX_PORT=1099
Environment=KAFKA_OPTS='-javaagent:/opt/jolokia-jvm-agent.jar'
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal
LimitNOFILE=65536

[Install]
WantedBy=default.target

CMAK error message:

2020-05-07 20:58:35,394 - [ERROR] - from kafka.manager.actor.cluster.BrokerViewCacheActor in kafka-manager-system-akka.actor.default-dispatcher-15
Failed to get broker metrics for BrokerIdentity(1,broker-1,1099,false,true,Map(PLAINTEXT -> 9092))
java.rmi.ConnectException: Connection refused to host: 127.0.1.1; nested exception is:
        java.net.ConnectException: Connection refused (Connection refused)
        at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:623)
        at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:209)
        at java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:196)
        at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:132)
        at java.management.rmi/javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
        at java.management.rmi/javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2105)
        at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:321)
        at java.management/javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
        at kafka.manager.jmx.KafkaJMX$.doWithConnection(KafkaJMX.scala:57)
        at kafka.manager.actor.cluster.BrokerViewCacheActor.$anonfun$updateBrokerMetrics$3(BrokerViewCacheActor.scala:359)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
        at scala.util.Success.$anonfun$map$1(Try.scala:255)
        at scala.util.Success.map(Try.scala:213)
        at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
        at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
        at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
        at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
        at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
        at java.base/java.net.Socket.connect(Socket.java:609)
        at java.base/java.net.Socket.connect(Socket.java:558)
        at java.base/java.net.Socket.<init>(Socket.java:454)
        at java.base/java.net.Socket.<init>(Socket.java:231)
        at java.rmi/sun.rmi.transport.tcp.TCPDirectSocketFactory.createSocket(TCPDirectSocketFactory.java:40)
        at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:617)
        ... 20 common frames omitted

[tanuj83]

same for me

[tanuj83]

It is working for me

marked cluster in CMAK as SSL and consumer.property with SSL config has been imported in application.conf

[tanuj83]

application.conf

kafka-manager.consumer.properties.file=${?CONSUMER_PROPERTIES_FILE} kafka-manager.consumer.properties.file="/etc/cmak/consumer.properties"

Consumer.properties

security.protocol=SSL ssl.truststore.location=server.truststore.jks ssl.truststore.password=XXXXXX ssl.keystore.location=server.keystore.jks ssl.keystore.password=XXXX ssl.key.password=XXXXX ssl.keystore.type=JKS ssl.truststore.type=JKS key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer