search

pinterest / DoctorK

DoctorK is a service for Kafka cluster auto healing and workload balancing
Apache License 2.0
633 stars 92 forks source link

java.lang.NoClassDefFoundError: scala/reflect/internal/util/Statistics$ #120

Open sidharthk opened 5 years ago

sidharthk commented 5 years ago

Hi

I have downloaded the doctorkafka from https://github.com/pinterest/doctorkafka.git and buid completed successfully. However while running KafkaStatMain command i get below error . Please help Command:

java -server \ -Dlog4j.configurationFile=file:./config/log4j2.xml \ -cp target/lib/:target//:target/:target/kafkastats-0.2.4.4.jar:/kafka/tools/doctorkafka/kafkastats/target/classes/com/pinterest/doctorkafka/*:target/kafkastats-0.2.4.4-jar-with-dependencies.jar \ com.pinterest.doctorkafka.stats.KafkaStatsMain \ -broker kafka-poc-4 \ -jmxport 8888 \ -topic brokerstats \ -zookeeper kafka-poc-4:2181/cluster \ -uptimeinseconds 3600 \ -pollingintervalinseconds 60 \ -ostrichport 2051 \ -tsdhostport localhost:18126 \ -kafka_config /kafka/server.properties \ -producer_config /kafka/producer.properties \ -primary_network_ifacename eth0

ERROR: [root@sidharth-kafka-poc-1 kafkastats]# sh dockafka.sh 2019-04-08 18:28:55,208 main DEBUG Apache Log4j Core 2.11.1 initializing configuration XmlConfiguration[location=/kafka/tools/doctorkafka/kafkastats/./config/log4j2.xml] 2019-04-08 18:28:55,414 main DEBUG Installed 2 script engines

Failed to initialize compiler: NoClassDefFoundError. This is most often remedied by a full clean and recompile. Otherwise, your classpath may continue bytecode compiled by different and incompatible versions of scala.

java.lang.NoClassDefFoundError: scala/reflect/internal/util/Statistics$ at scala.tools.nsc.symtab.SymbolLoaders$ClassfileLoader.doComplete(SymbolLoaders.scala:310) at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:213) at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1530) at scala.reflect.internal.Symbols$Symbol.initialize(Symbols.scala:1678) at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1422) at scala.tools.nsc.Global$Run.(Global.scala:1140) at scala.tools.nsc.interpreter.IMain._initialize(IMain.scala:127) at scala.tools.nsc.interpreter.IMain.initializeSynchronous(IMain.scala:149) at scala.tools.nsc.interpreter.Scripted.(Scripted.scala:74) at scala.tools.nsc.interpreter.Scripted$.apply(Scripted.scala:309) at scala.tools.nsc.interpreter.Scripted$Factory.getScriptEngine(Scripted.scala:302) at org.apache.logging.log4j.core.script.ScriptManager.(ScriptManager.java:99) at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:216) at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250) at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:619) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:636) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:231) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194) at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:581) at com.pinterest.doctorkafka.stats.KafkaStatsMain.(KafkaStatsMain.java:34) Caused by: java.lang.ClassNotFoundException: scala.reflect.internal.util.Statistics$ at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 23 more

Failed to initialize compiler: NoClassDefFoundError. This is most often remedied by a full clean and recompile. Otherwise, your classpath may continue bytecode compiled by different and incompatible versions of scala.

java.lang.NoClassDefFoundError: scala/reflect/internal/util/Statistics$ at scala.tools.nsc.symtab.SymbolLoaders$ClassfileLoader.doComplete(SymbolLoaders.scala:310) at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:213) at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1530) at scala.reflect.internal.Symbols$Symbol.initialize(Symbols.scala:1678) at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1422) at scala.tools.nsc.Global$Run.(Global.scala:1140) at scala.tools.nsc.interpreter.IMain._initialize(IMain.scala:127) at scala.tools.nsc.interpreter.IMain.global$lzycompute(IMain.scala:156) at scala.tools.nsc.interpreter.IMain.global(IMain.scala:155) at scala.tools.nsc.interpreter.IMain.initializeSynchronous(IMain.scala:150) at scala.tools.nsc.interpreter.Scripted.(Scripted.scala:74) at scala.tools.nsc.interpreter.Scripted$.apply(Scripted.scala:309) at scala.tools.nsc.interpreter.Scripted$Factory.getScriptEngine(Scripted.scala:302) at org.apache.logging.log4j.core.script.ScriptManager.(ScriptManager.java:99) at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:216) at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:250) at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:547) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:619) at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:636) at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:231) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153) at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45) at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194) at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:581) at com.pinterest.doctorkafka.stats.KafkaStatsMain.(KafkaStatsMain.java:34) SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Exception in thread "main" java.lang.NullPointerException at java.util.Hashtable.put(Hashtable.java:460) at com.pinterest.doctorkafka.util.OperatorUtil.createKafkaProducerProperties(OperatorUtil.java:234) at com.pinterest.doctorkafka.stats.KafkaAvroPublisher.(KafkaAvroPublisher.java:60) at com.pinterest.doctorkafka.stats.KafkaStatsMain.main(KafkaStatsMain.java:135) 2019-04-08 18:28:58,428 pool-1-thread-1 DEBUG Stopping LoggerContext[name=31befd9f, org.apache.logging.log4j.core.LoggerContext@6a988392]

sidharthk commented 5 years ago

Hi Yang,

Thanks for your reply.

Initially I just had target/lib/* and target/kafkastats-0.2.4.4.jar in classpath but due to error and for trail purpose I added other one after another hierarchy but nothing helped.

Warm Regards

Sidharth Kumar |

On Tue, 9 Apr, 2019, 6:02 PM Yu Yang, notifications@github.com wrote:

can you exclude /kafka/tools/doctorkafka/kafkastats/target/classes/com/pinterest/doctorkafka/* from the class path and see if that fixes the problem?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pinterest/doctorkafka/issues/120#issuecomment-481231687, or mute the thread https://github.com/notifications/unsubscribe-auth/Au8ZIaNcwTdUbP2JAZLIkVDJjpU61J6Uks5vfIhrgaJpZM4ciJ0r .

sidharthk commented 5 years ago

Any help please. I am not able to understand why this issues as I do see scala-reflect-2.12.x.jar in lib directory as well. Any input will be great help

yuyang08 commented 5 years ago

I am not sure if it is because of any issue in 0.2.4.4. I tried with the latest kafkastats binary, and it worked fine on our test cluster. Can you sync with the latest code and try again? 

java -server -Dlog4j.configurationFile=file:./config/log4j2.xml . \
  -cp target/lib/:target//:target/:target/kafkastats-0.2.4.5.jar:/kafka/tools/doctorkafka/kafkastats/target/classes/com/pinterest/doctorkafka/*:target/kafkastats-0.2.4.5-jar-with-dependencies.jar   \
  com.pinterest.doctorkafka.stats.KafkaStatsMain -jmxport 9999 \
  -topic brokerstats -zookeeper testzk001:2181/testkafka  \
  -tsdhostport localhost:18126 -ostrichport 2052 \
  -uptimeinseconds 3600 -pollingintervalinseconds 60 \
  -kafka_config /etc/kafka/server.properties \
  -broker testbroker001  \
  -producer_config  /etc/kafka/producer.properties \
  -primary_network_ifacename eth0
sidharthk commented 5 years ago

Thanks Yu Yang for your help.

I have Apache Kafka 1.0.2 deployed. So could you please confirm if latest Kafkastats supports 1.0.2 Kafka.

It will be great help if by any means you could share the binaries.

Warm Regards

Sidharth Kumar | Mob: +91 8197 555 599|

On Fri, 12 Apr, 2019, 11:54 AM Yu Yang, notifications@github.com wrote:

I am not sure if it is because of any issue in 0.2.4.4. I tried with the latest kafkastats binary, and it worked fine on our test cluster:

java -server -Dlog4j.configurationFile=file:./config/log4j2.xml . \ -cp target/lib/:target//:target/:target/kafkastats-0.2.4.5.jar:/kafka/tools/doctorkafka/kafkastats/target/classes/com/pinterest/doctorkafka/*:target/kafkastats-0.2.4.5-jar-with-dependencies.jar \ com.pinterest.doctorkafka.stats.KafkaStatsMain -jmxport 9999 \ -topic brokerstats -zookeeper testzk001:2181/testkafka \ -tsdhostport localhost:18126 -ostrichport 2052 \ -uptimeinseconds 3600 -pollingintervalinseconds 60 \ -kafka_config /etc/kafka/server.properties \ -broker testkafka080 \ -producer_config /etc/kafka/producer.properties \ -primary_network_ifacename eth0

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pinterest/doctorkafka/issues/120#issuecomment-482453319, or mute the thread https://github.com/notifications/unsubscribe-auth/Au8ZIf9u_vdcI0fAvh92TKUYs5VH_vqhks5vgCaggaJpZM4ciJ0r .

yuyang08 commented 5 years ago

I tested kafkastats 0.2.4.5 with kafka 1.0.2 binary (https://archive.apache.org/dist/kafka/1.0.2/kafka_2.12-1.0.2.tgz) , and it worked fine for me.

Note that kafkastats by default use scala 2.12 packages. If you have scala 2.11 installed on your local machine, that may cause jar conflicts.

sidharthk commented 5 years ago

Thanks for your help,build 0.2.4.5 resolved the issue. However i am getting another error now in 0.2.4.5 https://github.com/pinterest/doctorkafka/issues/128 . Please let me know where i am going wrong with configurations