storm-kafka and kafka_2.9.2 "0.8.x", "0.9.x" compatibility

[andrewmilkowski]

hello

seeking to run storm-kafka against newer kafka codebase (0.8.x, 0.9.x) codebease in contrib/storm-kafka is compatible with 0.7.x only..

any chance in the near term for a port to the newer kafka ?

thanks!

[asmaier]

This would be very useful indeed.

[jarretraim]

Is it not? The current dependencies for the project seem to state that it's working with newer storm versions:

:provided {:dependencies [[storm "0.9.0-wip15"]

[jasonjckn]

I think 0.7 and 0.8 kafka will be in use for a long time, so it makes sense to maintain two branches of storm-kafka for each respectively. It would be awesome if someone did a PR for this, I personally haven't needed 0.8 yet.

Is it not? The current dependencies for the project seem to state that it's working with newer storm versions: Andrew is referring to kafka version. Yes the latest storm-kafka works with 0.9 storm.

On Tue, Sep 24, 2013 at 5:56 AM, Jarret Raim notifications@github.comwrote:

Is it not? The current dependencies for the project seem to state that it's working with newer storm versions:

:provided {:dependencies [[storm "0.9.0-wip15"]

— Reply to this email directly or view it on GitHubhttps://github.com/nathanmarz/storm-contrib/issues/38#issuecomment-25000074 .

[asmaier]

See also here https://groups.google.com/forum/#!topic/storm-user/OXQ0a9ppcYM

[dschiavu]

Use this project for Kafka 0.8: https://github.com/wurstmeister/storm-kafka-0.8-plus/

[sujeetyadav]

Hi,

I am trying to integrate storm and kafka but I am facing issue such as I am able to deploy the topology but it is not listening to kafka messages.

Following are the maven dependencies I have used:

org.apache.storm storm-core 0.9.1-incubating

    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-kafka</artifactId>
        <version>0.9.2-incubating</version>
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.10</artifactId>
        <version>0.8.0</version>
        <exclusions>
            <exclusion>
                <groupId>org.apache.zookeeper</groupId>
                <artifactId>zookeeper</artifactId>
            </exclusion>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>0.8.2.1</version>
    </dependency>

When I run this program it gives following log: 4670 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded to master yet. Submitting jar... 4676 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar /home/sujeet/Downloads/fieldvantage/TimeSeriesDataMigration/target/TimeSeriesDataMigration-0.0.1-SNAPSHOT.jar to assigned location: /usr/local/storm/data/nimbus/inbox/stormjar-009d6969-b1fc-475a-8e3a-3d0b988fecdd.jar 4691 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded topology jar to assigned location: /usr/local/storm/data/nimbus/inbox/stormjar-009d6969-b1fc-475a-8e3a-3d0b988fecdd.jar 4691 [main] INFO backtype.storm.StormSubmitter - Submitting topology TestTopology in distributed mode with conf {"topology.workers":1,"topology.message.timeout.secs":600,"topology.debug":false,"kafka.broker.properties":{"bootstrap.servers":"localhost:9092","acks":"1","value.serializer":"org.apache.kafka.common.serialization.StringSerializer","key.serializer":"org.apache.kafka.common.serialization.StringSerializer"}}

But when I am sending messages from Kafka producer it is unable to listen it.

Any help would be appreciable.

Thanks & Regards, Sujeet Yadav

[Dennis1122]

Hi,

I too have the same issuewith kafka storm integration. I am not able to read from kafka topic when running on a single node cluster. Everything works fine while running in local mode. @sujeetyadav Are you able to solve the issue.. ??

Any help will be highly appreciated.

Thanks in advance Dennis