search

RuckusWirelessIL / pentaho-kafka-consumer

Apache Kafka consumer step plug-in for Pentaho Kettle
Apache License 2.0
66 stars 40 forks source link

Kafka 0.10 support is missing #14

Closed utish closed 7 years ago

utish commented 8 years ago

I am using Pentaho Data Integration 6.1 and trying to connect to Kafka (2.10-0.10.0.0). I download the consumer from here and put it in plugins/steps folder. While running, I'm getting this error:

2016/08/01 09:12:00 - Apache Kafka Consumer.0 - ERROR (version 6.1.0.1-196, build 1 from 2016-04-07 12.08.49 by buildguy) : Error initializing step [Apache Kafka Consumer] 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - ERROR (version 6.1.0.1-196, build 1 from 2016-04-07 12.08.49 by buildguy) : java.lang.NoClassDefFoundError: Could not initialize class org.apache.log4j.LogManager 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at org.apache.log4j.Logger.getLogger(Logger.java:104) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at kafka.utils.Logging$class.logger(Logging.scala:24) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at kafka.utils.VerifiableProperties.logger$lzycompute(VerifiableProperties.scala:26) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at kafka.utils.VerifiableProperties.logger(VerifiableProperties.scala:26) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at kafka.utils.Logging$class.info(Logging.scala:67) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at kafka.utils.VerifiableProperties.info(VerifiableProperties.scala:26) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at kafka.utils.VerifiableProperties.verify(VerifiableProperties.scala:217) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at kafka.consumer.ConsumerConfig.(ConsumerConfig.scala:95) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at com.ruckuswireless.pentaho.kafka.consumer.KafkaConsumerStep.init(KafkaConsumerStep.java:64) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at org.pentaho.di.trans.step.StepInitThread.run(StepInitThread.java:69) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at java.lang.Thread.run(Unknown Source) 2016/08/01 09:12:00 - kafkaConsumer - ERROR (version 6.1.0.1-196, build 1 from 2016-04-07 12.08.49 by buildguy) : Step [Apache Kafka Consumer.0] failed to initialize! 2016/08/01 09:12:00 - Spoon - ERROR (version 6.1.0.1-196, build 1 from 2016-04-07 12.08.49 by buildguy) : kafkaConsumer: preparing transformation execution failed 2016/08/01 09:12:00 - Spoon - ERROR (version 6.1.0.1-196, build 1 from 2016-04-07 12.08.49 by buildguy) : org.pentaho.di.core.exception.KettleException: 2016/08/01 09:12:00 - Spoon - We failed to initialize at least one step. Execution can not begin! 2016/08/01 09:12:00 - Spoon - 2016/08/01 09:12:00 - Spoon - 2016/08/01 09:12:00 - Spoon - at org.pentaho.di.trans.Trans.prepareExecution(Trans.java:1142) 2016/08/01 09:12:00 - Spoon - at org.pentaho.di.ui.spoon.trans.TransGraph$29.run(TransGraph.java:4035) 2016/08/01 09:12:00 - Spoon - at java.lang.Thread.run(Unknown Source)

Am I missing anything? Thanks.

spektom commented 8 years ago

Hi, This plugin supports Kafka 0.8.x. I guess we would need a different version for Kafka 0.10.x.

On Mon, Aug 1, 2016, 21:01 utish notifications@github.com wrote:

I am using Pentaho Data Integration 6.1 and trying to connect to Kafka (2.10-0.10.0.0). I download the consumer from here http://wiki.pentaho.com/display/EAI/Apache+Kafka+Consumer and put it in plugins/steps folder. While running, I'm getting this error:

2016/08/01 09:12:00 - Apache Kafka Consumer.0 - ERROR (version 6.1.0.1-196, build 1 from 2016-04-07 12.08.49 by buildguy) : Error initializing step [Apache Kafka Consumer] 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - ERROR (version 6.1.0.1-196, build 1 from 2016-04-07 12.08.49 by buildguy) : java.lang.NoClassDefFoundError: Could not initialize class org.apache.log4j.LogManager 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at org.apache.log4j.Logger.getLogger(Logger.java:104) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at kafka.utils.Logging$class.logger(Logging.scala:24) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at kafka.utils.VerifiableProperties.logger$lzycompute(VerifiableProperties.scala:26) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at kafka.utils.VerifiableProperties.logger(VerifiableProperties.scala:26) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at kafka.utils.Logging$ class.info(Logging.scala:67) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at kafka.utils.VerifiableProperties.info(VerifiableProperties.scala:26) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at kafka.utils.VerifiableProperties.verify(VerifiableProperties.scala:217) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at kafka.consumer.ConsumerConfig.(ConsumerConfig.scala:95) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at com.ruckuswireless.pentaho.kafka.consumer.KafkaConsumerStep.init(KafkaConsumerStep.java:64) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at org.pentaho.di.trans.step.StepInitThread.run(StepInitThread.java:69) 2016/08/01 09:12:00 - Apache Kafka Consumer.0 - at java.lang.Thread.run(Unknown Source) 2016/08/01 09:12:00 - kafkaConsumer - ERROR (version 6.1.0.1-196, build 1 from 2016-04-07 12.08.49 by buildguy) : Step [Apache Kafka Consumer.0] failed to initialize! 2016/08/01 09:12:00 - Spoon - ERROR (version 6.1.0.1-196, build 1 from 2016-04-07 12.08.49 by buildguy) : kafkaConsumer: preparing transformation execution failed 2016/08/01 09:12:00 - Spoon - ERROR (version 6.1.0.1-196, build 1 from 2016-04-07 12.08.49 by buildguy) : org.pentaho.di.core.exception.KettleException: 2016/08/01 09:12:00 - Spoon - We failed to initialize at least one step. Execution can not begin! 2016/08/01 09:12:00 - Spoon - 2016/08/01 09:12:00 - Spoon - 2016/08/01 09:12:00 - Spoon - at org.pentaho.di.trans.Trans.prepareExecution(Trans.java:1142) 2016/08/01 09:12:00 - Spoon - at org.pentaho.di.ui.spoon.trans.TransGraph$29.run(TransGraph.java:4035) 2016/08/01 09:12:00 - Spoon - at java.lang.Thread.run(Unknown Source)

Am I missing anything? Thanks.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/RuckusWirelessIL/pentaho-kafka-consumer/issues/14, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJpO65M1_lByguSSN0zmgL8CNPsCw8dks5qbjR8gaJpZM4JZ1jf .

utish commented 8 years ago

Also tested in Kafka 0.8.x and faced the same issue.

spektom commented 7 years ago

There's an option to build consumer plug-in for a specific Kafka version now (see README.md file for more information)