nathanmarz / storm-contrib

A collection of spouts, bolts, serializers, DSLs, and other goodies to use with Storm
Eclipse Public License 1.0
580 stars 342 forks source link

storm-kafka and kafka_2.9.2 "0.8.x", "0.9.x" compatibility #38

Open andrewmilkowski opened 11 years ago

andrewmilkowski commented 11 years ago

hello

seeking to run storm-kafka against newer kafka codebase (0.8.x, 0.9.x) codebease in contrib/storm-kafka is compatible with 0.7.x only..

any chance in the near term for a port to the newer kafka ?

thanks!

asmaier commented 11 years ago

This would be very useful indeed.

jarretraim commented 11 years ago

Is it not? The current dependencies for the project seem to state that it's working with newer storm versions:

:provided {:dependencies [[storm "0.9.0-wip15"]

jasonjckn commented 11 years ago

I think 0.7 and 0.8 kafka will be in use for a long time, so it makes sense to maintain two branches of storm-kafka for each respectively. It would be awesome if someone did a PR for this, I personally haven't needed 0.8 yet.

Is it not? The current dependencies for the project seem to state that it's working with newer storm versions: Andrew is referring to kafka version. Yes the latest storm-kafka works with 0.9 storm.

On Tue, Sep 24, 2013 at 5:56 AM, Jarret Raim notifications@github.comwrote:

Is it not? The current dependencies for the project seem to state that it's working with newer storm versions:

:provided {:dependencies [[storm "0.9.0-wip15"]

— Reply to this email directly or view it on GitHubhttps://github.com/nathanmarz/storm-contrib/issues/38#issuecomment-25000074 .

asmaier commented 11 years ago

See also here https://groups.google.com/forum/#!topic/storm-user/OXQ0a9ppcYM

dschiavu commented 10 years ago

Use this project for Kafka 0.8: https://github.com/wurstmeister/storm-kafka-0.8-plus/

sujeetyadav commented 8 years ago

Hi,

I am trying to integrate storm and kafka but I am facing issue such as I am able to deploy the topology but it is not listening to kafka messages.

Following are the maven dependencies I have used:

org.apache.storm storm-core 0.9.1-incubating
    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-kafka</artifactId>
        <version>0.9.2-incubating</version>
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.10</artifactId>
        <version>0.8.0</version>
        <exclusions>
            <exclusion>
                <groupId>org.apache.zookeeper</groupId>
                <artifactId>zookeeper</artifactId>
            </exclusion>
            <exclusion>
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>0.8.2.1</version>
    </dependency>

When I run this program it gives following log: 4670 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded to master yet. Submitting jar... 4676 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar /home/sujeet/Downloads/fieldvantage/TimeSeriesDataMigration/target/TimeSeriesDataMigration-0.0.1-SNAPSHOT.jar to assigned location: /usr/local/storm/data/nimbus/inbox/stormjar-009d6969-b1fc-475a-8e3a-3d0b988fecdd.jar 4691 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded topology jar to assigned location: /usr/local/storm/data/nimbus/inbox/stormjar-009d6969-b1fc-475a-8e3a-3d0b988fecdd.jar 4691 [main] INFO backtype.storm.StormSubmitter - Submitting topology TestTopology in distributed mode with conf {"topology.workers":1,"topology.message.timeout.secs":600,"topology.debug":false,"kafka.broker.properties":{"bootstrap.servers":"localhost:9092","acks":"1","value.serializer":"org.apache.kafka.common.serialization.StringSerializer","key.serializer":"org.apache.kafka.common.serialization.StringSerializer"}}

But when I am sending messages from Kafka producer it is unable to listen it.

Any help would be appreciable.

Thanks & Regards, Sujeet Yadav

Dennis1122 commented 7 years ago

Hi,

I too have the same issuewith kafka storm integration. I am not able to read from kafka topic when running on a single node cluster. Everything works fine while running in local mode. @sujeetyadav Are you able to solve the issue.. ??

Any help will be highly appreciated.

Thanks in advance Dennis