Closed nownikhil closed 3 years ago
small note:
Rebasing every time makes reviewers have to look through the entire commit again since we can't see what changed.
We use squash merge at the end, so there is no benefit to rebasing in terms of the final commit history. Would you be willing to just add commits so we can more easily review?
Yeah that makes sense, my bad. Will add a new commit next time.
In this change we are adding support for
HashCoGroup
andCoGroupedPipe
. For evaluating HashCoGroup we are creating a ParDo transformation on the larger pipe with smaller pipe as side input. For evaluating CoGroupedPipe we are using theMultiJoinFunction
to evaluate the final output iterator.Also as part of this change we are doing a minor refactor for code > 100 lines.
TESTS: Added unit tests for both HashCoGroup and CoGroupedPipe.