Adding Beam-Dataflow support for group functions in Scalding - Githubissues

twitter / scalding

A Scala API for Cascading

http://twitter.com/scalding

Apache License 2.0

3.48k stars 703 forks source link

Adding Beam-Dataflow support for group functions in Scalding #1942

Closed Franco1010 closed 2 years ago

Franco1010 commented 2 years ago

This PR adds support for scalding group functions by handling:

Grouped
- IdentityReduce
UnsortedGrouped
- UnsortedIdentityReduce
- IteratorMappedReduce
SortedGrouped
- IdentityValueSortedReduce
- ValueSortedReduce

via

mapGroup
sortedMapGroup
sorted

methods in BeamOp.

Optimized sum functions in mapGroup through planMapGroup
Functions using priorityQueue are not supported in this PR, because they mutate elements passed as input, but Beam does not allow to mutate input between transformations