search

RuckusWirelessIL / pentaho-kafka-consumer

Apache Kafka consumer step plug-in for Pentaho Kettle
Apache License 2.0
66 stars 40 forks source link

kafka 2.11.010 err #21

Closed hote123456 closed 7 years ago

hote123456 commented 7 years ago

producer kafka_2.11-0.10.0.0,using consumer default lib,can't received anything。 message is ：2017-04-20 14:42:51 [IFF:56]-[INFO] topicName:BI_LOG JSONObjectString={"appid":"SMS","channel":"UP"} config as this Topic name：BI_LOG Target key field name：message Target key field name：a

if changed lib kafka_2.10-0.10.0.0.jar 、kafka-clients-0.10.0.0.jar ，running has error log 2017/04/20 15:26:19 - Spoon - Transformation opened. 2017/04/20 15:26:19 - Spoon - Launching transformation [pisp]... 2017/04/20 15:26:19 - Spoon - Started the transformation execution. 2017/04/20 15:26:19 - pisp - Dispatching started for transformation [pisp] 2017/04/20 15:26:20 - Apache Kafka Consumer.0 - Creating Kafka consumer listening on zookeeper: 10.121.30.90:2181 2017/04/20 15:26:31 - Apache Kafka Consumer.0 - ERROR (version 7.0.0.0-25, build 1 from 2016-11-05 15.35.36 by buildguy) : Error initializing step [Apache Kafka Consumer] 2017/04/20 15:26:31 - Apache Kafka Consumer.0 - ERROR (version 7.0.0.0-25, build 1 from 2016-11-05 15.35.36 by buildguy) : kafka.common.ConsumerRebalanceFailedException: group_team-PC-1492673180099-4ac86678 can't rebalance after 4 retries 2017/04/20 15:26:31 - Apache Kafka Consumer.0 - at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:670) 2017/04/20 15:26:31 - Apache Kafka Consumer.0 - at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:977) 2017/04/20 15:26:31 - Apache Kafka Consumer.0 - at kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:264) 2017/04/20 15:26:31 - Apache Kafka Consumer.0 - at kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:85) 2017/04/20 15:26:31 - Apache Kafka Consumer.0 - at kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:97) 2017/04/20 15:26:31 - Apache Kafka Consumer.0 - at com.ruckuswireless.pentaho.kafka.consumer.KafkaConsumerStep.init(KafkaConsumerStep.java:71) 2017/04/20 15:26:31 - Apache Kafka Consumer.0 - at org.pentaho.di.trans.step.StepInitThread.run(StepInitThread.java:69) 2017/04/20 15:26:31 - Apache Kafka Consumer.0 - at java.lang.Thread.run(Thread.java:745) 2017/04/20 15:26:31 - pisp - ERROR (version 7.0.0.0-25, build 1 from 2016-11-05 15.35.36 by buildguy) : Step [Apache Kafka Consumer.0] failed to initialize! 2017/04/20 15:26:31 - Spoon - ERROR (version 7.0.0.0-25, build 1 from 2016-11-05 15.35.36 by buildguy) : pisp: preparing transformation execution failed 2017/04/20 15:26:31 - Spoon - ERROR (version 7.0.0.0-25, build 1 from 2016-11-05 15.35.36 by buildguy) : org.pentaho.di.core.exception.KettleException: 2017/04/20 15:26:31 - Spoon - We failed to initialize at least one step. Execution can not begin! 2017/04/20 15:26:31 - Spoon - 2017/04/20 15:26:31 - Spoon - 2017/04/20 15:26:31 - Spoon - at org.pentaho.di.trans.Trans.prepareExecution(Trans.java:1144) 2017/04/20 15:26:31 - Spoon - at org.pentaho.di.ui.spoon.trans.TransGraph$29.run(TransGraph.java:4156) 2017/04/20 15:26:31 - Spoon - at java.lang.Thread.run(Thread.java:745)

hote123456 commented 7 years ago

detail log for default log 2017/04/20 16:00:28 - Text file output.0 - Opening output stream in default encoding 2017/04/20 16:00:28 - Text file output.0 - Opened new file with name [file:///C:/Users/eking/Desktop/file.txt] 2017/04/20 16:00:28 - Apache Kafka Consumer.0 - Released server socket on port 0 2017/04/20 16:00:28 - Apache Kafka Consumer.0 - Creating Kafka consumer listening on zookeeper: 10.121.30.90:2181 2017/04/20 16:00:29 - Apache Kafka Consumer.0 - Received streams map: {BI_LOG=[group kafka stream]} 2017/04/20 16:00:29 - pisp - Step [Apache Kafka Consumer.0] initialized flawlessly. 2017/04/20 16:00:29 - pisp - Step [Text file output.0] initialized flawlessly. 2017/04/20 16:00:29 - Apache Kafka Consumer.0 - Starting to run... 2017/04/20 16:00:29 - Apache Kafka Consumer.0 - Starting message consumption with overall timeout of 0ms 2017/04/20 16:00:29 - Apache Kafka Consumer.0 - Starting direct consumption 2017/04/20 16:00:29 - Apache Kafka Consumer.0 - Collecting unlimited messages 2017/04/20 16:00:29 - Text file output.0 - Starting to run... 2017/04/20 16:00:29 - pisp - Transformation has allocated 2 threads and 1 rowsets. 2017/04/20 16:00:39 - pisp - Triggering heartbeat signal for pisp at every 10 seconds 2017/04/20 16:00:49 - pisp - Triggering heartbeat signal for pisp at every 10 seconds

spektom commented 7 years ago

Do you see other exceptions related to Kafka re-balancing issue? This doesn't look like an exception related to PDI. It could happen, for instance, if you're using 'roundrobin' partition assigner, and your consumer group is already used with a different set of topics.

spektom commented 7 years ago

Actually, it seems to be a duplicate of #7