twitter / summingbird

Streaming MapReduce with Scalding and Storm
https://twitter.com/summingbird
Apache License 2.0
2.14k stars 267 forks source link

Add tests for flatMapValues/mapValues/etc... #639

Open johnynek opened 8 years ago

johnynek commented 8 years ago

Memory did not support it, so clearly we are not exercising it.

See note in: #638

johnynek commented 8 years ago

There actually are tests:

https://github.com/twitter/summingbird/blob/develop/summingbird-core-test/src/main/scala/com/twitter/summingbird/TestGraphs.scala#L194

We are just not running those tests in the Memory/ConcurrentMemory platforms.

They may not work correctly because I guess there is a hidden assumption that there is no pipelining for the logic of that to be correct (for each key, you have to push it all the way through before you can process the next value for that key). As such, we need to detect those nodes and combine them into one operation to handle it correctly (something storm still does not do, as of this writing).