twitter / summingbird

Streaming MapReduce with Scalding and Storm
https://twitter.com/summingbird
Apache License 2.0
2.14k stars 267 forks source link

Upgrade to Storm API 1.0.2 #699

Closed ttim closed 7 years ago

ttim commented 7 years ago

Support modern Storm API.


Can be merged after chill, scalding and tormenta's releases and bumping their versions.

johnynek commented 7 years ago

what are the plans on this branch @ttim ?

ttim commented 7 years ago

Updated this PR: now using Storm's completeTopology to avoid Thread.sleep.

@johnynek I'm planning to merge it after tormenta's release.

ttim commented 7 years ago

@johnynek Travis is green after I disabled mima check. I'm unsure should I merge this change or not, because even it's green it contains two versions of kryo at the same time (I don't think it breaks anything in most cases, but still). From other point it would be a little bit more convenient for me to have this branch merged. What do you think?

johnynek commented 7 years ago

Yeah, maybe it is good to merge this when we scalding with kryo 0.8.3 On Tue, Jan 17, 2017 at 16:53 Timur notifications@github.com wrote:

@johnynek https://github.com/johnynek Travis is green after I disabled mima check. I'm unsure should I merge this change or not, because even it's green it contains two versions of kryo at the same time (I don't think it breaks anything in most cases, but still). From other point it would be a little bit more convenient for me to have this branch merged. What do you think?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/twitter/summingbird/pull/699#issuecomment-273368687, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEJdqDmIg1kHa58onXmKn2zt2A-ma-fks5rTX68gaJpZM4KvWsn .

codecov-io commented 7 years ago

Codecov Report

Merging #699 into develop will increase coverage by 0.11%. The diff coverage is 87.5%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #699      +/-   ##
===========================================
+ Coverage    70.85%   70.96%   +0.11%     
===========================================
  Files          141      141              
  Lines         3479     3475       -4     
  Branches       202      201       -1     
===========================================
+ Hits          2465     2466       +1     
+ Misses        1014     1009       -5
Impacted Files Coverage Δ
...witter/summingbird/storm/spout/KeyValueSpout.scala 77.77% <ø> (ø) :arrow_up:
.../com/twitter/summingbird/storm/StormPlatform.scala 82.35% <ø> (ø) :arrow_up:
...witter/summingbird/storm/FlatMapBoltProvider.scala 100% <ø> (ø) :arrow_up:
...la/com/twitter/summingbird/storm/StormMetric.scala 0% <ø> (ø) :arrow_up:
...rd/storm/collector/AggregatorOutputCollector.scala 90.9% <ø> (ø) :arrow_up:
...tter/summingbird/storm/option/FlatMapOptions.scala 50% <ø> (ø) :arrow_up:
.../twitter/summingbird/storm/StormStatProvider.scala 92.85% <ø> (ø) :arrow_up:
...ter/summingbird/storm/spout/TraversableSpout.scala 100% <ø> (ø) :arrow_up:
.../twitter/summingbird/storm/StoreStatReporter.scala 0% <ø> (ø) :arrow_up:
...scala/com/twitter/summingbird/storm/BaseBolt.scala 77.1% <ø> (ø) :arrow_up:
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update d4965e5...5eaa49b. Read the comment docs.

ttim commented 7 years ago

@johnynek We released scalding so I think it's time to merge this PR (as soon as tests will pass). How does it sound for you?

After that I'm going to make one more PR (https://github.com/ttim/summingbird/tree/tabishev/bump_scalding_algebird_versions) which contains @piyushnarang changes for algebird (https://github.com/twitter/summingbird/pull/715) as well as scalding version bump.

After that as soon as @pankajroark will release new storehaus version we'll be able to do SB release, I guess with scala 2.12 support as well.

johnynek commented 7 years ago

Yeah, I think that all sounds good. On Wed, May 10, 2017 at 14:00 Timur notifications@github.com wrote:

@johnynek https://github.com/johnynek We released scalding so I think it's time to merge this PR (as soon as tests will pass). How does it sound for you?

After that I'm going to make one more PR ( https://github.com/ttim/summingbird/tree/tabishev/bump_scalding_algebird_versions) which contains @piyushnarang https://github.com/piyushnarang changes for algebird (#715 https://github.com/twitter/summingbird/pull/715) as well as scalding version bump.

After that as soon as @pankajroark https://github.com/pankajroark will release new storehaus version we'll be able to do SB release, I guess with scala 2.12 support as well.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/twitter/summingbird/pull/699#issuecomment-300645243, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEJdlqSdRX2WVLkhkJ-QorzpRp2QYBSks5r4k-pgaJpZM4KvWsn .