search

druid-io / tranquility

Tranquility helps you send real-time event streams to Druid and handles partitioning, replication, service discovery, and schema rollover, seamlessly and without downtime.
Apache License 2.0
515 stars 230 forks source link

Support for Apache Storm 1.0 #154

Open saurabhere opened 8 years ago

saurabhere commented 8 years ago

With the 1.0 release of Apache Storm, many APIs have been moved. See Commit: https://github.com/apache/storm/commit/d839d1bf88b855edda344fc548f0701e2a018655

Due to this, any Storm Topologies with tranquility beams are rejected with ClassNotFoundExceptions. Need to update the dependencies in the tranquility-storm library.

gianm commented 8 years ago

@saurabhere do you know if it is possible for a single tranquility-storm module to work with pre- and post-1.0? in other words: do we need both tranquility-storm and tranquility-storm1 modules?

(we don't want to drop support for storm pre-1.0)

saurabhere commented 8 years ago

The change would be a non-backwards compatible one. We have 2 choices if we want to support 0.x releases:

  1. Compile a separate tranquility-storm jar for 1.0
  2. Defer the change until most users are on 1.0 and suggest the hack discussed in the JIRA ticket until then. Basically, adding 
client.jartransformer.class: "org.apache.storm.hack.StormShadeTransformer"

to the storm.yaml would allow code compiled for old versions to work on 1.0.0. We should document this though.

a1nayak commented 7 years ago

Tried using the transformer class but throws error as below. Believe tranquility needs to be recompiled using 1.x storm java.lang.NoSuchMethodError: org.apache.storm.spout.MultiScheme.deserialize([B)Ljava/lang/Iterable; at storm.kafka.KafkaUtils.generateTuples(KafkaUtils.java:206) ~[stormjar.jar:?] at storm.kafka.trident.TridentKafkaEmitter.emit(TridentKafkaEmitter.java:183) ~[stormjar.jar:?] at storm.kafka.trident.TridentKafkaEmitter.doEmitNewPartitionBatch(TridentKafkaEmitter.java:124) ~[stormjar.jar:?] at storm.kafka.trident.TridentKafkaEmitter.failFastEmitNewPartitionBatch(TridentKafkaEmitter.java:70) ~[stormjar.jar:?] at storm.kafka.trident.TridentKafkaEmitter.emitNewPartitionBatch(TridentKafkaEmitter.java:77) ~[stormjar.jar:?] at storm.kafka.trident.TridentKafkaEmitter.access$000(TridentKafkaEmitter.java:44) ~[stormjar.jar:?] at storm.kafka.trident.TridentKafkaEmitter$1.emitPartitionBatch(TridentKafkaEmitter.java:217) ~[stormjar.jar:?] at storm.kafka.trident.TridentKafkaEmitter$1.emitPartitionBatch(TridentKafkaEmitter.java:207) ~[stormjar.jar:?] at org.apache.storm.trident.spout.OpaquePartitionedTridentSpoutExecutor$Emitter.emitBatch(OpaquePartitionedTridentSpoutExecutor.java:128) ~[storm-core-1.0.2.jar:1.0.2] at org.apache.storm.trident.spout.TridentSpoutExecutor.execute(TridentSpoutExecutor.java:82) ~[storm-core-1.0.2.jar:1.0.2] at org.apache.storm.trident.topology.TridentBoltExecutor.execute(TridentBoltExecutor.java:383) ~[storm-core-1.0.2.jar:1.0.2] at org.apache.storm.daemon.executor$fn__8058$tuple_action_fn__8060.invoke(executor.clj:731) ~[storm-core-1.0.2.jar:1.0.2] at org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464) ~[storm-core-1.0.2.jar:1.0.2] at org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40) ~[storm-core-1.0.2.jar:1.0.2] at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) ~[storm-core-1.0.2.jar:1.0.2] at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) ~[storm-core-1.0.2.jar:1.0.2] at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) ~[storm-core-1.0.2.jar:1.0.2] at org.apache.storm.daemon.executor$fn__8058$fn__8071$fn__8124.invoke(executor.clj:850) ~[storm-core-1.0.2.jar:1.0.2] at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484) [storm-core-1.0.2.jar:1.0.2] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]

westerly commented 7 years ago

Hi guys! What's the status on this issue? Are you planning to support Storm 1.0 any time soon?

joanvr commented 7 years ago

Hi! We also need support for Storm 1.0. Any news?

fjy commented 7 years ago

Hey guys, we've been a bit backlogged on PRs reviews. Will get to this soon.

On Tue, Dec 13, 2016 at 6:33 AM, Joan Viladrosa notifications@github.com wrote:

Hi! We also need support for Storm 1.0. Any news?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/druid-io/tranquility/issues/154#issuecomment-266753314, or mute the thread https://github.com/notifications/unsubscribe-auth/AAaJJR2Sna4OloSXZLoS2HDoKtvSh7M-ks5rHqylgaJpZM4IHq3f .

yuusaku-t commented 7 years ago

Is there anything that I can help with? I would like to use tranquility on Storm 1.0.x (Recently, Storm 1.0.3 has been released).

yuusaku-t commented 7 years ago

I found the following library in the storm-official repository. https://github.com/apache/storm/tree/master/external/storm-druid https://issues.apache.org/jira/browse/STORM-1979

Will the codes of storm in druid-io/tranquility repository be deprecated in the future?