Open ashishsoni opened 9 years ago
Having the same issue. It doesn't seem to be able to determine it, and displays it as -1.
I DID set a JMX_PORT when running my kafka server.
@ashishsoni You need to export JMX_PORT, before starting your broker. Check Kafka documentation.
@shermany The JMX port used by the broker is published in zookeeper, if you did set it, it was not picked up. Verify using netstat the port is actually listening by the broker process on your brokers.
@patelh Ahhh yes, I dun goofed. Was the end of a long day and I had just added JMX_PORT into kafka-server-start.sh (hacky), and didn't export it either. So of course it wasn't actually picked up.
Set the env properly in my upstart script that manages kafka and all was good.
Is the JMX_PORT thing actually in the kafka docs though? I only found it through forum posts and what not via google and then looking at the scripts themselves.
@ashishsoni @shermany You can start the kafka cluster by adding JMX_PORT, and I solved the problem by using this command, but the broker metric info is not updated. $JMX_PORT=8004 ./kafka-server-start.sh ../config/server.properties &
There are no error or warning on the server but even after sending many messages on broker, metric is not getting updated
[INFO] [06/11/2015 13:04:29.427] [kafka-manager-system-akka.actor.default-dispatcher-3] [akka://kafka-manager-system/user/kafka-manager/172.16.1.13/broker-view] Updating broker view...
@HarvinderBhullar did you set JMX_PORT before starting kafka broker? If yes, do you see the JMX port when you look at list of brokers? If yes, did you enable JMX polling in the cluster config via the "Modify" button in the cluster list for that cluster?
@patelh. Yes, the JMX_PORT is set before starting the kafka. I see the JMX_PORT on the broker list. The JMX_POLLING is also enabled. Just pasted the data from the screen below.
Id Host Port JMX Port Bytes In Bytes Out
0 172.16.1.42 9092 9997 0.00 0.00
@patelh. I upgraded to 8.2 and everything is working fine as expected.
one can also add the following line to bin/kafka-server-start.sh.
export JMX_PORT=${JMX_PORT:-9999}
What is accessing port 9999? Is it the server that is running kafkamanager? Or is it the localhost from where I'm running my web browser?
env JMX_PORT=9999 bin/kafka-server-start.sh /Users/JaideepsinhGohil/Apache/kafka_2.11-0.8.2.2/config/server.properties restart kafka manager and upgrade to 8.2 in Modify cluster menu
Hi everyone, I set the JMX_PORT=9999 but still have some errors. So my metrics are all zeros
[error] k.m.j.KafkaJMX$ - Failed to connect to service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) ~[na:1.8.0_31]
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[na:1.8.0_31]
at kafka.manager.jmx.KafkaJMX$.doWithConnection(KafkaJMX.scala:57) ~[kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply$mcV$sp(BrokerViewCacheActor.scala:326) [kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply(BrokerViewCacheActor.scala:323) [kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply(BrokerViewCacheActor.scala:323) [kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) [org.scala-lang.scala-library-2.11.8.jar:na]
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) [org.scala-lang.scala-library-2.11.8.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_31]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_31]
Caused by: javax.naming.CommunicationException: null
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:122) ~[na:1.8.0_31]
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205) ~[na:1.8.0_31]
at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[na:1.8.0_31]
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1929) ~[na:1.8.0_31]
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1896) ~[na:1.8.0_31]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[na:1.8.0_31]
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[na:1.8.0_31]
at kafka.manager.jmx.KafkaJMX$.doWithConnection(KafkaJMX.scala:57) ~[kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply$mcV$sp(BrokerViewCacheActor.scala:326) [kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply(BrokerViewCacheActor.scala:323) [kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:304) ~[na:1.8.0_31]
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.8.0_31]
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342) ~[na:1.8.0_31]
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source) ~[na:1.8.0_31]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118) ~[na:1.8.0_31]
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205) ~[na:1.8.0_31]
at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[na:1.8.0_31]
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1929) ~[na:1.8.0_31]
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1896) ~[na:1.8.0_31]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[na:1.8.0_31]
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:980) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1363) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:735) ~[na:1.8.0_31]
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) ~[na:1.8.0_31]
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.8.0_31]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[na:1.8.0_31]
at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[na:1.8.0_31]
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:229) ~[na:1.8.0_31]
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.8.0_31]
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342) ~[na:1.8.0_31]
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:961) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1363) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:735) ~[na:1.8.0_31]
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) ~[na:1.8.0_31]
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.8.0_31]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[na:1.8.0_31]
at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[na:1.8.0_31]
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:229) ~[na:1.8.0_31]
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.8.0_31]
and
[error] k.m.a.c.BrokerViewCacheActor - Failed to get topic metrics for broker BrokerIdentity(0,localhost,9092,9999,false)
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) ~[na:1.8.0_31]
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[na:1.8.0_31]
at kafka.manager.jmx.KafkaJMX$.doWithConnection(KafkaJMX.scala:57) ~[kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply$mcV$sp(BrokerViewCacheActor.scala:326) ~[kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply(BrokerViewCacheActor.scala:323) ~[kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply(BrokerViewCacheActor.scala:323) ~[kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[org.scala-lang.scala-library-2.11.8.jar:na]
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[org.scala-lang.scala-library-2.11.8.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_31]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_31]
Caused by: javax.naming.CommunicationException: null
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:122) ~[na:1.8.0_31]
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205) ~[na:1.8.0_31]
at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[na:1.8.0_31]
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1929) ~[na:1.8.0_31]
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1896) ~[na:1.8.0_31]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[na:1.8.0_31]
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[na:1.8.0_31]
at kafka.manager.jmx.KafkaJMX$.doWithConnection(KafkaJMX.scala:57) ~[kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply$mcV$sp(BrokerViewCacheActor.scala:326) ~[kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateTopicMetrics$1$$anonfun$apply$24$$anonfun$apply$25$$anonfun$apply$2.apply(BrokerViewCacheActor.scala:323) ~[kafka-manager.kafka-manager-1.3.2.1-sans-externalized.jar:na]
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:304) ~[na:1.8.0_31]
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.8.0_31]
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342) ~[na:1.8.0_31]
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source) ~[na:1.8.0_31]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118) ~[na:1.8.0_31]
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205) ~[na:1.8.0_31]
at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[na:1.8.0_31]
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1929) ~[na:1.8.0_31]
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1896) ~[na:1.8.0_31]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[na:1.8.0_31]
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:980) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1363) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:735) ~[na:1.8.0_31]
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) ~[na:1.8.0_31]
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.8.0_31]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[na:1.8.0_31]
at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[na:1.8.0_31]
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:229) ~[na:1.8.0_31]
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.8.0_31]
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342) ~[na:1.8.0_31]
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:961) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1363) ~[na:1.8.0_31]
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:735) ~[na:1.8.0_31]
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) ~[na:1.8.0_31]
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.8.0_31]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[na:1.8.0_31]
at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[na:1.8.0_31]
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:229) ~[na:1.8.0_31]
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.8.0_31]
I am running the entire setup on windows.. I made the changes to built kafka-manager for windows and run it successfully. But when i enabled the JMX on my brokers and restart each kafka broker node with its respective JMX port, the kafka manager gave these errors
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40) ~[na:1.8.0_40] at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148) ~[na:1.8.0_40] at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613) ~[na:1.8.0_40]
[error] k.m.a.c.BrokerViewCacheActor - Failed to get broker metrics for BrokerIdentity(2,192.168.1.4,9095,9992,false) java.rmi.UnknownHostException: Unknown host: 127.0.0.1:2181; nested exception is: java.net.UnknownHostException: 127.0.0.1:2181 at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:616) ~[na:1.8.0_40] at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) ~[na:1.8.0_40] at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.8.0_40] at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130) ~[na:1.8.0_40] at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source) ~[na:1.8.0_40] at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2432) ~[na:1.8.0_40] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308) ~[na:1.8.0_40] at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[na:1.8.0_40] at kafka.manager.jmx.KafkaJMX$.doWithConnection(KafkaJMX.scala:57) ~[kafka-manager.kafka-manager-1.3.3.6-sans-externalized.jar:na] at kafka.manager.actor.cluster.BrokerViewCacheActor$$anonfun$kafka$manager$actor$cluster$BrokerViewCacheActor$$updateBrokerMetrics$1$$anonfun$apply$27$$anonfun$apply$3.apply$mcV$sp(BrokerViewCacheActor.scala:358) ~[kafka-manager.kafka-manager-1.3.3.6-sans-externalized.jar:na] Caused by: java.net.UnknownHostException: 127.0.0.1:2181 at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) ~[na:1.8.0_40] at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) ~[na:1.8.0_40] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_40] at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_40] at java.net.Socket.connect(Socket.java:538) ~[na:1.8.0_40] at java.net.Socket.
(Socket.java:434) ~[na:1.8.0_40] at java.net.Socket. (Socket.java:211) ~[na:1.8.0_40] at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40) ~[na:1.8.0_40] at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148) ~[na:1.8.0_40] at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613) ~[na:1.8.0_40]
If you enable JMX it is only available on the same host, if you want to access it from the kafka-manager server you have to configure it accordingly:
If you use a systemd service
:
-Environment=JMX_PORT=5556
+Environment="KAFKA_JMX_OPTS=-Djava.rmi.server.hostname=10.0.0.1 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=5556 -Dcom.sun.management.jmxremote.port=5556 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
If you use a run-script:
-export JMX_PORT=5556
+export KAFKA_JMX_OPTS="-Djava.rmi.server.hostname=10.0.0.1 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.rmi.port=5556 -Dcom.sun.management.jmxremote.port=5556 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
(Replace 10.0.0.1
with the kafka-server's ip)
Copied from SO and https://github.com/yahoo/kafka-manager/issues/186
@ST-DDT thanks, worked for me.
Where do i specify the JMX port ?