search

twitter / summingbird

Streaming MapReduce with Scalding and Storm
https://twitter.com/summingbird
Apache License 2.0
2.14k stars 267 forks source link

Issue #675.Configurable elimination of FlatMapNode by enhancing Sourc… #676

Closed NPraneeth closed 8 years ago

NPraneeth commented 8 years ago

…eNode

Closes #675 In some topologies, FlatMapNode is just created to handle the formatting for the SummerNode. This feature tries to eliminate the FlatMapNode by enhancing the SourceNode to handle the formatting logic compatible to summer. This also means that the aggregation will happen only in SummerNode. The enhancements to the SourceNode include bringing the flatMap operation into SourceNode, new formatting of SourceNode when followed by SummerNode. A new spout and output collector are implemented to handle the formatting logic. This feature is also made configurable via Options, which are taken into consideration right from the planning phase for Online.

johnynek commented 8 years ago

Okay, just a couple of minor changes then ready to merge.

Question: any benchmarks or measurements from tests of this?

johnynek commented 8 years ago

👍

jnievelt commented 8 years ago

Minor readability comments; looks good to me otherwise.

pankajroark commented 8 years ago

Looks good to me.

jnievelt commented 8 years ago

🚢

johnynek commented 8 years ago

Went ahead and merged.

Good work on this @NPraneeth !

sriramkrishnan commented 8 years ago

Yup, what @johnynek said. Thanks @NPraneeth!

NPraneeth commented 8 years ago

Thanks @johnynek @sriramkrishnan. Looking forward to contribute more.