wadhwasam
commented
6 years ago Any update on this issue?
Closed wadhwasam closed 6 years ago
wadhwasam
commented
6 years ago Any update on this issue?
Ycallaer
commented
6 years ago Hi,
Ran into the same issue
[info] k.m.a.c.OffsetCachePassive - Starting consumers path children cache... [error] k.m.a.c.package$TryLogErrorHelper - Failed to create consumer for offset topic for cluster KafkaDev java.util.NoSuchElementException: key not found: PLAINTEXT at scala.collection.MapLike$class.default(MapLike.scala:228) ~[org.scala-lang.scala-library-2.11.8.jar:na] at scala.collection.AbstractMap.default(Map.scala:59) ~[org.scala-lang.scala-library-2.11.8.jar:na] at scala.collection.MapLike$class.apply(MapLike.scala:141) ~[org.scala-lang.scala-library-2.11.8.jar:na] at scala.collection.AbstractMap.apply(Map.scala:59) ~[org.scala-lang.scala-library-2.11.8.jar:na] at kafka.manager.actor.cluster.KafkaManagedOffsetCache$$anonfun$14.apply(KafkaStateActor.scala:195) ~[kafka-manager.kafka-manager-1.3.3.17-sans-externalized.jar:na] at kafka.manager.actor.cluster.KafkaManagedOffsetCache$$anonfun$14.apply(KafkaStateActor.scala:194) ~[kafka-manager.kafka-manager-1.3.3.17-sans-externalized.jar:na] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) ~[org.scala-lang.scala-library-2.11.8.jar:na] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) ~[org.scala-lang.scala-library-2.11.8.jar:na] at scala.collection.Iterator$class.foreach(Iterator.scala:893) ~[org.scala-lang.scala-library-2.11.8.jar:na] at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) ~[org.scala-lang.scala-library-2.11.8.jar:na]
I configured the consumer.properties file. The manager starts partially but lots of issues in the background.
lllewellyn
commented
6 years ago After looking at this a bit, I don't think the consumer.properties file has any effect. I wanted to use SASL_SSL. I've successfully configured other clients to connect to Kafka with just a few properties, leaving others at defaults:
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
ssl.truststore.location=/path/to/jks/file
ssl.truststore.password=theSecret
ssl.keystore.location=/path/to/jks/file
ssl.keystore.password=theSecretSo, I tried adding these to consumer.properties. But when I look at the log, the only properties shown when a consumer is created are those provided in via the 'create cluster' screen in the app.
It looks like the ultimate fix is to modify multiple code files, allowing the additional parameters to be supplied by the user and then provided to consumer creation code. Here are a few I think need to be changed:
I think the changes would be to allow configuration of the legal SaslConfig and SslConfig properties from Kafka, then pass these on to consumers when they are created.
Note, I am neither a Scala nor a Play guru so I may be missing or misunderstanding things. I mention my findings in case someone with more expertise wants to take a crack at making an enhancement. I didn't look at any existing PRs to see if it's already in progress.
matagyula
commented
6 years ago Hello, I too am facing the same issue. The 3 node Kafka cluster I'm administering has both SASL and SSL endpoints exposed, I'm using the SASL_PLAINTEXT mode for Kafka Manager (it has it's own principal and keytab, and the accompanying JAAS config file). Even though the error message is being spammed in the logs, Kafka Manager itself seems to be working - I can see topic details on the Web UI, broker info, etc. On another cluster I have an older version of Kafka Manager (kafka-manager-1.3.3.14), with both SASL and SSL enabled, and that one is not complaining about a missing PLAINTEXT key. (using SASL_PLAINTEXT mode, pretty much in the same way I have version 1.3.3.17 setup). To me it seems like an unnecessary error message that's clogging up the logs 😕
cemo
commented
6 years ago https://gist.github.com/cemo/6e7f4607d3b9abe66ffe08ee9c428761
Since I could not find any solution I had to suppress log on my cluster. I have disabled logger for this exception until it will be resolved since I could not any solution.
matagyula
commented
6 years ago Thanks for the tip, cemo. If I get this right, your solution turns off logging completely for Kafka Manager?
cemo
commented
6 years ago @matagyula no actually. Just kafka.manager.model logger.
matagyula
commented
6 years ago @cemo - looking at your code I manually edited my logback.xml file and I've added the line <logger name="kafka.manager.model" level"OFF" /> to it, and the kafka manager logfile stopped getting entries altogether :x Did I miss something?
cemo
commented
6 years ago It means don't log any logger starting with kafka.manager.model.*. Other loggers will continue.
[info] k.m.a.c.BrokerViewCacheActor - Updating broker view...
[info] k.m.a.KafkaManagerActor - Updating internal state...
[info] k.m.a.KafkaManagerActor - Updating internal state...
[info] k.m.a.KafkaManagerActor - Updating internal state...
[info] k.m.a.c.BrokerViewCacheActor - Updating broker view...
[info] k.m.a.KafkaManagerActor - Updating internal state...
[info] k.m.a.KafkaManagerActor - Updating internal state...
[info] k.m.a.KafkaManagerActor - Updating internal state...
[info] k.m.a.c.BrokerViewCacheActor - Updating broker view...
[info] k.m.a.KafkaManagerActor - Updating internal state...
[info] k.m.a.KafkaManagerActor - Updating internal state...
[info] k.m.a.KafkaManagerActor - Updating internal state...
[info] k.m.a.c.BrokerViewCacheActor - Updating broker view...
[info] k.m.a.KafkaManagerActor - Updating internal state...Here is a simple output from our log. Actually we are not interested in kafka-manager's logging output since we are using it very basic.
matagyula
commented
6 years ago Thank you for the info, cemo! I just noticed a typo in my config, I forgot the equal sign in front of the "OFF" command, heh. No wonder I had no logs whatsoever :) I modified the config to logger name="kafka.manager.actor.cluster.KafkaStateActor" level="OFF", now it's no longer logging the PLAINTEXT error messages. Thanks again!
patelh
commented
6 years ago Upgrade to 1.3.3.18 for 1.0+ support. SSL support is already there, just configure the consumer.properties file and set the cluster config to use SSL. We have it working in our SSL enabled clusters.
Leon-WTF
commented
6 years ago @patelh I'am using kafka-manager-1.3.3.20 and kafka_2.11-1.1.0(With only SSL enabled), and I'm still facing this issue? Any suggestion? consumer.properties: security.protocol=SSL key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer ssl.truststore.location=/path/to/client.truststore.jks ssl.truststore.password=pass ssl.keystore.location=/path/to/client.keystore.jks ssl.keystore.password=pass ssl.key.password=pass
commands: cd /home/tengfei/kafka-manager/target/universal/kafka-manager-1.3.3.20 nohup bin/kafka-manager -Dconfig.file=./conf/application.conf -java-home /usr/share/jdk1.8.0_161/ -Dhttp.port=9001 &
Error: 2018-07-24 22:14:09,288 - [ERROR] - from kafka.manager.actor.cluster.KafkaStateActor in kafka-manager-system-akka.actor.default-dispatcher-10 KafkaTopicOffsetGetter exception java.util.NoSuchElementException: key not found: PLAINTEXT at scala.collection.MapLike$class.default(MapLike.scala:228) ~[org.scala-lang.scala-library-2.11.12.jar:na] at scala.collection.AbstractMap.default(Map.scala:59) ~[org.scala-lang.scala-library-2.11.12.jar:na] at scala.collection.MapLike$class.apply(MapLike.scala:141) ~[org.scala-lang.scala-library-2.11.12.jar:na] at scala.collection.AbstractMap.apply(Map.scala:59) ~[org.scala-lang.scala-library-2.11.12.jar:na] at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter$$anonfun$run$7$$anonfun$apply$10$$anonfun$apply$mcV$sp$14.apply(KafkaStateActor.scala:1459) ~[kafka-manager.kafka-manager-1.3.3.20-sans-externalized.jar:na] at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter$$anonfun$run$7$$anonfun$apply$10$$anonfun$apply$mcV$sp$14.apply(KafkaStateActor.scala:1451) ~[kafka-manager.kafka-manager-1.3.3.20-sans-externalized.jar:na] at scala.collection.Iterator$class.foreach(Iterator.scala:891) ~[org.scala-lang.scala-library-2.11.12.jar:na] at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) ~[org.scala-lang.scala-library-2.11.12.jar:na] at scala.collection.MapLike$DefaultKeySet.foreach(MapLike.scala:174) ~[org.scala-lang.scala-library-2.11.12.jar:na] at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter$$anonfun$run$7$$anonfun$apply$10.apply$mcV$sp(KafkaStateActor.scala:1451) ~[kafka-manager.kafka-manager-1.3.3.20-sans-externalized.jar:na] at scala.util.control.Breaks.breakable(Breaks.scala:38) ~[org.scala-lang.scala-library-2.11.12.jar:na] at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter$$anonfun$run$7.apply(KafkaStateActor.scala:1450) ~[kafka-manager.kafka-manager-1.3.3.20-sans-externalized.jar:na] at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter$$anonfun$run$7.apply(KafkaStateActor.scala:1419) ~[kafka-manager.kafka-manager-1.3.3.20-sans-externalized.jar:na] at scala.Option.fold(Option.scala:158) ~[org.scala-lang.scala-library-2.11.12.jar:na] at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter.run(KafkaStateActor.scala:1419) ~[kafka-manager.kafka-manager-1.3.3.20-sans-externalized.jar:na] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161]
jia2
commented
6 years ago @Leon-WTF I had the same issue. It seems that not all parameters defined in consumer.properties are used when constructing KafkaClient. I resolved this by passing ssl related parameters to JVM, (like -Djavax.net.ssl.trustStore=/path/to/my.jks -Djavax.net.ssl.trustStorePassword=mypassword ).
jsargiot
commented
6 years ago I can confirm that version 1.3.3.21 works fine with SSL only clusters using consumer.properties file only (without jvm parameters).
@Leon-WTF, was you cluster configured in Kafka Manager as SSL before? or was PLAINTEXT and then you change it to SSL? If so, I had to restart KM after changing it to SSL to work.
bthodla
commented
6 years ago Hello, I too am facing the same issue. The 3 node Kafka cluster I'm administering has both SASL and SSL endpoints exposed, I'm using the SASL_PLAINTEXT mode for Kafka Manager (it has it's own principal and keytab, and the accompanying JAAS config file). Even though the error message is being spammed in the logs, Kafka Manager itself seems to be working - I can see topic details on the Web UI, broker info, etc. On another cluster I have an older version of Kafka Manager (kafka-manager-1.3.3.14), with both SASL and SSL enabled, and that one is not complaining about a missing PLAINTEXT key. (using SASL_PLAINTEXT mode, pretty much in the same way I have version 1.3.3.17 setup). To me it seems like an unnecessary error message that's clogging up the logs
I get the same error but in my case, the web UI fails to come up
ADMarcelo
commented
3 years ago vi config/server.properties
My kafka cluster is SSL enabled, I started kafka manger in a separate box. Everything seems to be working fine except I am getting the Key not found error as show below. Please let me know how to fix this issue
[error] k.m.a.c.KafkaStateActor - KafkaTopicOffsetGetter exception java.util.NoSuchElementException: key not found: PLAINTEXT at scala.collection.MapLike$class.default(MapLike.scala:228) ~[org.scala-lang.scala-library-2.11.8.jar:na] at scala.collection.AbstractMap.default(Map.scala:59) ~[org.scala-lang.scala-library-2.11.8.jar:na] at scala.collection.MapLike$class.apply(MapLike.scala:141) ~[org.scala-lang.scala-library-2.11.8.jar:na] at scala.collection.AbstractMap.apply(Map.scala:59) ~[org.scala-lang.scala-library-2.11.8.jar:na] at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter$$anonfun$run$7$$anonfun$apply$11$$anonfun$apply$mcV$sp$14.apply(KafkaStateActor.scala:1389) ~[kafka-manager.kafka-manager-1.3.3.17-sans-externalized.jar:na] at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter$$anonfun$run$7$$anonfun$apply$11$$anonfun$apply$mcV$sp$14.apply(KafkaStateActor.scala:1383) ~[kafka-manager.kafka-manager-1.3.3.17-sans-externalized.jar:na] at scala.collection.Iterator$class.foreach(Iterator.scala:893) ~[org.scala-lang.scala-library-2.11.8.jar:na] at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) ~[org.scala-lang.scala-library-2.11.8.jar:na] at scala.collection.MapLike$DefaultKeySet.foreach(MapLike.scala:174) ~[org.scala-lang.scala-library-2.11.8.jar:na] at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter$$anonfun$run$7$$anonfun$apply$11.apply$mcV$sp(KafkaStateActor.scala:1383) ~[kafka-manager.kafka-manager-1.3.3.17-sans-externalized.jar:na]