search

yahoo / CMAK

CMAK is a tool for managing Apache Kafka clusters
Apache License 2.0
11.83k stars 2.51k forks source link

Broker Metric not getting updated #69

Open ashishsoni opened 9 years ago

ashishsoni commented 9 years ago

Messages in /sec 0.00 0.00 0.00 0.00 Bytes in /sec 0.00 0.00 0.00 0.00 Bytes out /sec 0.00 0.00 0.00 0.00 Bytes rejected /sec 0.00 0.00 0.00 0.00 Failed fetch request /sec 0.00 0.00 0.00 0.00 Failed produce request /sec 0.00 0.00 0.00 0.00

patelh commented 9 years ago

You need to set JMX_PORT for your broker before starting it.

ashishsoni commented 9 years ago

I did setup the JMX port but still not getting updated

guangliangChen commented 9 years ago

It is the same to me,and No JMX Port Error was disappeared after I add JMX port. Do anyone knows the reason.

style95 commented 9 years ago

I am also unable to have metrics updated. I found "javax.management.InstanceNotFoundException" from the server log.

So I browsed JMX metrics using my own Java program and found ObjectName is correct. What I found is ObjectName should include double quotes in values and domain.

javax.management.InstanceNotFoundException: kafka.server:type=BrokerTopicMetrics,name=sample-service-MessagesInPerSec
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095) ~[na:1.7.0_79]
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttributes(DefaultMBeanServerInterceptor.java:679) ~[na:1.7.0_79]
    at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttributes(JmxMBeanServer.java:705) ~[na:1.7.0_79]
    at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468) ~[na:1.7.0_79]
    at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97) ~[na:1.7.0_79]

For example, it should be like this: invalid : "kafka.server:type=BrokerTopicMetrics,name=sample-service-MessagesInPerSec" correct : "\"kafka.server\":type=\"BrokerTopicMetrics\",name=\"sample-service-MessagesInPerSec\""

If I use above invalid form, I am getting same error: "InstanceNotFoundException". Maybe Kafka-manager source code needs to be updated.

ystory commented 9 years ago

Same problem here

[debug] s.r.client.call - pool-14-thread-21: outbound call received exception: [xxx.xxx.xxx.xxx:xxxxx] exception:
javax.management.InstanceNotFoundException: kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=xxxxx
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095) ~[na:1.8.0_40]
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttributes(DefaultMBeanServerInterceptor.java:679) ~[na:1.8.0_40]
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttributes(JmxMBeanServer.java:705) ~[na:1.8.0_40]
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1447) ~[na:1.8.0_40]
        at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76) ~[na:1.8.0_40]
patelh commented 9 years ago

@style95 what version of Java are you using and what version of Kafka?

patelh commented 9 years ago

@ashishsoni @ystory @guangliangChen what version of Java are you using with Kafka? What version of Kafka are you running?

ashishsoni commented 9 years ago

Please see below

java version "1.7.0_79" OpenJDK Runtime Environment (IcedTea 2.5.5) (7u79-2.5.5-0ubuntu0.14.04.2) OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)

Kafka

kafka_2.9.1-0.8.2.1

Ashish

ystory commented 9 years ago

@patelh java version "1.8.0_40" Java(TM) SE Runtime Environment (build 1.8.0_40-b26) Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)

kafka_2.8.0-0.8.1.1

config /kafka-manager/configs/dev={"name":"dev","curatorConfig":{"host:2181/dev-kafka","zkMaxRetry":100,"baseSleepTimeMs":100,"maxSleepTimeMs":1000},"enabled":false,"kafkaVersion":"0.8.1.1","jmxEnabled":true}

Thank you. :)

style95 commented 9 years ago

java version "1.7.0_79" OpenJDK Runtime Environment (rhel-2.5.5.3.el6_6-x86_64 u79-b14) OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)

kafka_2.9.2-0.8.1.1 running on CenOS 6.5

Thank you : )

Dongkyoung

rycoxiao commented 9 years ago

I also can't update the metrics data. But i have any error information in server.log.

patelh commented 9 years ago

@rycoxiao what version of Java and Kafka are you using?

patelh commented 9 years ago

Going to address this by getting it working with OpenJDK.

rycoxiao commented 9 years ago

@patelh Linux version 3.13.0-24-generic (buildd@panlong) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014

java: java version "1.7.0_71" Java(TM) SE Runtime Environment (build 1.7.0_71-b14) Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode)

kafka: 0.8.1.1

untitled

imcom commented 9 years ago

same here, I am have the following setup

kafka 2.10 0.8.1.1 Linux ip-172-31-21-168 3.13.0-36-generic #63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

java version "1.7.0_75" OpenJDK Runtime Environment (IcedTea 2.5.4) (7u75-2.5.4-1~trusty1) OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode)

yyguzhou commented 9 years ago

I also can't get metrics updating, and can't get any error in server.log

centos 6.2 x86_64 kafka 2.9.2-0.8.1.1, use openjdk7u21 kafka-manager use openjdk8u45

HarvinderBhullar commented 9 years ago

I am facing the same issue, no error logs

java version "1.8.0_45"

terciodemelo commented 9 years ago

I've got the same issue, also with no error logs, has anyone made any progress on solving this? I intend to tackle it asap.

kafka 2.9.2-0.8.1.1 java version "1.8.0_31" Java(TM) SE Runtime Environment (build 1.8.0_31-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)

patelh commented 9 years ago

We use Oracle JDK and collecting JMX metrics seems to work fine. This may be an issue with OpenJDK. As @style95 pointed out, the JMX entity we query for seems to be incorrect for OpenJDK. The fix would just be detecting which JDK is being run and select the right format based on that. I haven't had time to do this yet.

yyguzhou commented 9 years ago

@patelh what's version?

ghost commented 9 years ago

same issue here.

kafka_2.9.2-0.8.1.1 on CentOS release 6.6 (Final) java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

patelh commented 9 years ago

Looks like mbean names changed between 0.8.1.1 and 0.8.2.* and we are currently using the 0.8.2 names.

patelh commented 9 years ago

Can someone try build from this branch and see if they get broker metrics with kafka 0.8.1.1 : #83

ghost commented 9 years ago

@patelh Thanks. It works for me.

KurtYoung commented 9 years ago

same problem here

java version: java version "1.7.0_51" Java(TM) SE Runtime Environment (build 1.7.0_51-b13) OpenJDK (Alibaba) 64-Bit Server VM (build 24.45-b08-internal, mixed mode)

kafka version: 2.10-0.8.2.1

linux version: Linux 2.6.32-220.23.2.ali927.el5.x86_64 #1 SMP Mon Jan 28 14:57:06 CST 2013 x86_64 x86_64 x86_64 GNU/Linux

patelh commented 9 years ago

I've merged changes to allow broker metrics for kafka 0.8.1.1 and 0.8.2.*. Please build off master.