search

RuckusWirelessIL / pentaho-kafka-consumer

Apache Kafka consumer step plug-in for Pentaho Kettle
Apache License 2.0
66 stars 40 forks source link

Some questions #11

Closed wgorman closed 7 years ago

wgorman commented 8 years ago

Thank you for contributing this step to the marketplace! I am writing up a blog entry as we speak demonstrating both the consumer and producer you've created. I have a few questions and wanted to see if I've missed something or if this all makes sense.

When I first run the plugins published on the marketplace, I get the following exception:

2015/12/23 11:58:18 - Apache Kafka Consumer.0 - ERROR (version 6.0.0.0-353, build 1 from 2015-10-07 13.27.43 by buildguy) : Error initializing step [Apache Kafka Consumer] 2015/12/23 11:58:18 - Apache Kafka Consumer.0 - ERROR (version 6.0.0.0-353, build 1 from 2015-10-07 13.27.43 by buildguy) : java.lang.NoClassDefFoundError: kafka/consumer/ConsumerConfig

Is this expected? To resolve the issue I placed all the lib/jar files in Spoon's main classpath. Do you have another approach? I am running Kettle 6.0 on Windows.

After I resolved that issue, I wasn't able to get the consumer to read any messages from Kafka. I tried setting various properties like auto.offset.reset=smallest, but I was unable to get anything out of Kafka. Also the transformation would stop quickly after running it. I am running Kafka 0.9 so maybe that had something to do with it.

To resolve that issue I checked out the latest source and updated the plugin to include the latest code. I noticed the new feature "Stop on empty topic", so figured there were changes that might be good to try out.

With the latest update, I still needed to copy the lib jars into the main classpath, after doing that I was able to successfully run the step!

Also with those changes I was able to get the producer working as well, I didn't need to get the latest version of the producer from github, the marketplace version worked against Kafka 0.9 just fine.

Would it be possible to update the marketplace with the latest version of the plugin?

Thank you again for contributing these steps!

Will

wgorman commented 8 years ago

Here's my writeup, thanks again for sharing these steps with the community! http://www.willgorman.com/?p=61

spektom commented 8 years ago

Hi Will,

Thank you for you interest and for the great article! I've just pushed an update to the Marketplace repository, but it will take some time until it's available on the Marketplace itself.

The problem you describe regarding the need to copy jar files into the Kettle's lib/ directory is weird, I've never encountered it before. I guess it has something to do with the version of Kettle that you're using. I'll try to run consumer/producer plug-ins with Kettle 6 soon.

Thanks! Michael

ghayes commented 8 years ago

I noticed in the "plugin.xml" file for the Consumer Plugin that you had lib/kafka_2.10-0.8.1.jar in the xml file, but lib/kafka_2.10-0.8.1.1.jar in the actual folder. Notice the extra ".1" at the end of the file name. Correcting the entry in plugin.xml fixed this for me. However, as with others, I am having trouble getting data using the Consumer plugin. I'll keep an eye out for the updates in the Marketplace.

Thanks, Greg

spektom commented 8 years ago

New plugin version depends on kafka_2.10-0.8.2.1.jar Can you grab the latest version?

Thanks, Michael

On Mon, Jan 4, 2016 at 11:16 PM Greg Hayes notifications@github.com wrote:

I noticed in the "plugin.xml" file for the Consumer Plugin that you had lib/kafka_2.10-0.8.1.jar in the xml file, but lib/kafka_2.10-0.8.1.1.jar in the actual folder. Notice the extra ".1" at the end of the file name. Correcting the entry in plugin.xml fixed this for me. However, as with others, I am having trouble getting data using the Consumer plugin. I'll keep an eye out for the updates in the Marketplace.

Thanks, Greg

— Reply to this email directly or view it on GitHub https://github.com/RuckusWirelessIL/pentaho-kafka-consumer/issues/11#issuecomment-168811210 .

wgorman commented 8 years ago

Thanks for updating the marketplace! I just tried out the latest version in my environment. I am still encountering the jar issue, maybe this is a windows only issue? Also, there now is a conflict with the xml-apis in Kettle 6.0, Kettle ships with version 2.0.2 of the jar and the plugin has an older version. Once I commented out the older version from the plugin.xml the consumer worked. The producer doesn't have an issue with it even though the jar is included in that as well.

spektom commented 7 years ago

Please try the newer version, it uses annotation-based registration.