twitter / summingbird

Streaming MapReduce with Scalding and Storm
https://twitter.com/summingbird
Apache License 2.0
2.14k stars 267 forks source link

Aggregate message IDs with values in AggregatorOutputCollector #693

Closed jnievelt closed 7 years ago

jnievelt commented 7 years ago

Fixes #692

Getting rid of streamMessageIdTracker because that's the wrong scope to track message IDs. Pushing the message IDs instead into the AsyncSummer.

Reworking TestAggregateOutpoutCollector a bit to accept expected message IDs and stream names. Existing tests just use None for these.

Adding new AggregatorOutputCollectorTest. This is pretty simple, using an AsyncSummer that always returns the same thing from addAll, but is sufficient to illustrate the issue captured in #692.

johnynek commented 7 years ago

:+1: