Error while fetching metadata with correlation id 12 : {__consumer_offsets=UNKNOWN}

[algorizm]

Thank for the great app! We're using Kafka 2.11-0.10 and spark 2.2.0, so I am running kafka-offset-monitor like:

java -cp KafkaOffsetMonitor-assembly-0.4.6-SNAPSHOT.jar com.quantifind.kafka.offsetapp.OffsetGetterWeb --offsetStorage kafka --kafkaBrokers kafka:9092,mongo:9092 --zk kafka:2181,mongo:2181 --port 9090 --ref resh 10.seconds --retain 2.days --dbName offsetappjava

When I start kafka-offset-monitor,I have the following error: 2018-03-02 01:32:15 INFO ConsumerConfig:165 - ConsumerConfig values: metric.reporters = [] metadata.max.age.ms = 300000 value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer group.id = kafka-monitor-LogEndOffsetGetter partition.assignment.strategy = [org.apache.kafka.clients.consumer.RangeAssignor] reconnect.backoff.ms = 50 sasl.kerberos.ticket.renew.window.factor = 0.8 max.partition.fetch.bytes = 1048576 bootstrap.servers = [kafka:9092, mongo:9092] retry.backoff.ms = 100 sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 ssl.keystore.type = JKS ssl.trustmanager.algorithm = PKIX enable.auto.commit = false ssl.key.password = null fetch.max.wait.ms = 500 sasl.kerberos.min.time.before.relogin = 60000 connections.max.idle.ms = 540000 ssl.truststore.password = null session.timeout.ms = 30000 metrics.num.samples = 2 client.id = ssl.endpoint.identification.algorithm = null key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer ssl.protocol = TLS check.crcs = true request.timeout.ms = 40000 ssl.provider = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.keystore.location = null heartbeat.interval.ms = 3000 auto.commit.interval.ms = 5000 receive.buffer.bytes = 32768 ssl.cipher.suites = null ssl.truststore.type = JKS security.protocol = PLAINTEXT ssl.truststore.location = null ssl.keystore.password = null ssl.keymanager.algorithm = SunX509 metrics.sample.window.ms = 30000 fetch.min.bytes = 1 send.buffer.bytes = 131072 auto.offset.reset = latest

2018-03-02 01:32:15 INFO KafkaOffsetGetter$:68 - Created admin client: kafka.admin.AdminClient@6e3f7ab4 2018-03-02 01:32:15 INFO AppInfoParser:82 - Kafka version : 0.9.0.1 2018-03-02 01:32:15 INFO AppInfoParser:83 - Kafka commitId : 23c69d62a0cabf06 2018-03-02 01:32:15 INFO AppInfoParser:82 - Kafka version : 0.9.0.1 2018-03-02 01:32:15 INFO AppInfoParser:83 - Kafka commitId : 23c69d62a0cabf06 2018-03-02 01:32:16 WARN NetworkClient:582 - Error while fetching metadata with correlation id 3 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:16 WARN NetworkClient:582 - Error while fetching metadata with correlation id 4 : {__consumer_offsets=UNKNOWN} 2018-03-02 01:32:16 WARN NetworkClient:582 - Error while fetching metadata with correlation id 5 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:16 WARN NetworkClient:582 - Error while fetching metadata with correlation id 6 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:16 WARN NetworkClient:582 - Error while fetching metadata with correlation id 7 : {__consumer_offsets=UNKNOWN} 2018-03-02 01:32:16 WARN NetworkClient:582 - Error while fetching metadata with correlation id 8 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:16 WARN NetworkClient:582 - Error while fetching metadata with correlation id 9 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:17 WARN NetworkClient:582 - Error while fetching metadata with correlation id 10 : {__consumer_offsets=UNKNOWN} 2018-03-02 01:32:17 WARN NetworkClient:582 - Error while fetching metadata with correlation id 11 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:17 WARN NetworkClient:582 - Error while fetching metadata with correlation id 12 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:17 WARN NetworkClient:582 - Error while fetching metadata with correlation id 13 : {__consumer_offsets=UNKNOWN} 2018-03-02 01:32:17 WARN NetworkClient:582 - Error while fetching metadata with correlation id 14 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:17 WARN NetworkClient:582 - Error while fetching metadata with correlation id 15 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:17 WARN NetworkClient:582 - Error while fetching metadata with correlation id 16 : {__consumer_offsets=UNKNOWN} 2018-03-02 01:32:17 WARN NetworkClient:582 - Error while fetching metadata with correlation id 17 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:17 WARN NetworkClient:582 - Error while fetching metadata with correlation id 18 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:17 WARN NetworkClient:582 - Error while fetching metadata with correlation id 19 : {__consumer_offsets=UNKNOWN} 2018-03-02 01:32:18 WARN NetworkClient:582 - Error while fetching metadata with correlation id 20 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:18 WARN NetworkClient:582 - Error while fetching metadata with correlation id 21 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:18 WARN NetworkClient:582 - Error while fetching metadata with correlation id 22 : {__consumer_offsets=UNKNOWN} 2018-03-02 01:32:18 WARN NetworkClient:582 - Error while fetching metadata with correlation id 23 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:18 WARN NetworkClient:582 - Error while fetching metadata with correlation id 24 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:18 WARN NetworkClient:582 - Error while fetching metadata with correlation id 25 : {__consumer_offsets=UNKNOWN} 2018-03-02 01:32:18 WARN NetworkClient:582 - Error while fetching metadata with correlation id 26 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:18 WARN NetworkClient:582 - Error while fetching metadata with correlation id 27 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:18 WARN NetworkClient:582 - Error while fetching metadata with correlation id 28 : {__consumer_offsets=UNKNOWN} 2018-03-02 01:32:19 WARN NetworkClient:582 - Error while fetching metadata with correlation id 29 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:19 WARN NetworkClient:582 - Error while fetching metadata with correlation id 30 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:19 WARN NetworkClient:582 - Error while fetching metadata with correlation id 31 : {__consumer_offsets=UNKNOWN} 2018-03-02 01:32:19 WARN NetworkClient:582 - Error while fetching metadata with correlation id 32 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:19 WARN NetworkClient:582 - Error while fetching metadata with correlation id 33 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:19 WARN NetworkClient:582 - Error while fetching metadata with correlation id 34 : {__consumer_offsets=UNKNOWN} 2018-03-02 01:32:19 WARN NetworkClient:582 - Error while fetching metadata with correlation id 35 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:19 WARN NetworkClient:582 - Error while fetching metadata with correlation id 36 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:19 WARN NetworkClient:582 - Error while fetching metadata with correlation id 37 : {__consumer_offsets=UNKNOWN} 2018-03-02 01:32:19 WARN NetworkClient:582 - Error while fetching metadata with correlation id 38 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:20 WARN NetworkClient:582 - Error while fetching metadata with correlation id 39 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:20 WARN NetworkClient:582 - Error while fetching metadata with correlation id 40 : {__consumer_offsets=UNKNOWN} 2018-03-02 01:32:20 WARN NetworkClient:582 - Error while fetching metadata with correlation id 41 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:20 WARN NetworkClient:582 - Error while fetching metadata with correlation id 42 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:20 WARN NetworkClient:582 - Error while fetching metadata with correlation id 43 : {__consumer_offsets=UNKNOWN} 2018-03-02 01:32:20 WARN NetworkClient:582 - Error while fetching metadata with correlation id 44 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:20 WARN NetworkClient:582 - Error while fetching metadata with correlation id 45 : {consumer_offsets=UNKNOWN} 2018-03-02 01:32:20 WARN NetworkClient:582 - Error while fetching metadata with correlation id 46 : {__consumer_offsets=UNKNOWN} 2018-03-02 01:32:20 WARN NetworkClient:582 - Error while fetching metadata with correlation id 47 : {consumer_offsets=UNKNOWN}

How can I fix that? Thanks.

[rcasey212]

Hi @algorizm:

This is caused because the monitor is not able to connect to the broker. This can be caused by a couple of reasons.

First: Test that the "kafka" and "mongo" domain names resolve to the correct brokers, and that the port is open to the machine that the monitor is running on.

Second: It could be related to the advertised.listeners and port setting in the server.properties file on your brokers. What could happen is that the consumer is trying to request METADATA for a given topic, figures out its advertised.listener and port and tries to connect. If these settings are not configured correctly it then may think that the topic is unavailable. Since you are using PLAINTEXT, your server.properties file should have lines that look something like:

advertised.listeners=PLAINTEXT://fully.qualified.domain.name:9092 port=9092

Bob

[algorizm]

Hi @rcasey212 👍 thank for reply.

I saw kafaka monitor web page in chrome browser. but when I select toplic list menu raise error, I have the following error: 2018-03-06 01:47:50 INFO ClientCnxn:1235 - Session establishment complete on server kafka/172.22.92.250:2181, sessionid = 0x161f50ce39b0010, negotiated timeout = 30000 2018-03-06 01:47:50 INFO ZkClient:711 - zookeeper state changed (SyncConnected) 2018-03-06 01:47:51 INFO OffsetGetter$:68 - Created zkUtils object: kafka.utils.ZkUtils@2216b57b 2018-03-06 01:47:58 WARN ServletHandler:586 - Error for /topiclist scala.NotImplementedError: an implementation is missing at scala.Predef$.$qmark$qmark$qmark(Predef.scala:230) at com.quantifind.kafka.core.ZKOffsetGetter.getTopicsAndLogEndOffsets(ZKOffsetGetter.scala:126) at com.quantifind.kafka.offsetapp.OffsetGetterWeb$$anonfun$getTopicsAndLogEndOffsets$1.apply(OffsetGetterWeb.scala:136) at com.quantifind.kafka.offsetapp.OffsetGetterWeb$$anonfun$getTopicsAndLogEndOffsets$1.apply(OffsetGetterWeb.scala:136) at com.quantifind.kafka.offsetapp.OffsetGetterWeb$.withOG(OffsetGetterWeb.scala:109) at com.quantifind.kafka.offsetapp.OffsetGetterWeb$.getTopicsAndLogEndOffsets(OffsetGetterWeb.scala:135) at com.quantifind.kafka.offsetapp.OffsetGetterWeb$$anon$1$$anonfun$intent$1.applyOrElse(OffsetGetterWeb.scala:190) at com.quantifind.kafka.offsetapp.OffsetGetterWeb$$anon$1$$anonfun$intent$1.applyOrElse(OffsetGetterWeb.scala:178) at scala.PartialFunction$Lifted.apply(PartialFunction.scala:223) at scala.PartialFunction$Lifted.apply(PartialFunction.scala:219) at unfiltered.response.Pass$PartialAttempt.attemptWithPass(pass.scala:72) at unfiltered.response.Pass$PassingAttempt$class.attempt(pass.scala:65) at unfiltered.response.Pass$PartialAttempt.attempt(pass.scala:67) at unfiltered.response.Pass$$anonfun$fold$1.apply(pass.scala:48) at unfiltered.response.Pass$FunctionAttempt.apply(pass.scala:77) at unfiltered.filter.Plan$class.doFilter(plans.scala:49) at com.quantifind.kafka.offsetapp.OffsetGetterWeb$$anon$1.doFilter(OffsetGetterWeb.scala:170) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1465) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.BlockingHttpConnection.handleReques

[rcasey212]

Hi @algorizm,

This is answered here: https://github.com/Morningstar/kafka-offset-monitor/issues/20

Bob