search

twitter / summingbird

Streaming MapReduce with Scalding and Storm
https://twitter.com/summingbird
Apache License 2.0
2.14k stars 267 forks source link

Externalizing optionMap collapse and metrics registration for StormPlatform #668

Closed jnievelt closed 8 years ago

jnievelt commented 8 years ago

For mapping bolts, we generally use FlatMapOperation which wraps the mapping function in an Externalizer. We don't currently do this for functions that go directly into the spout, however.

This change wraps both the optionMap function and the registerAllMetrics function in Externalizers before giving them to the spout.

This allows, for example, using Source#collect with Storm (i.e., pushing a partial function into the spout).

johnynek commented 8 years ago

looks good to me other than some small comments.

johnynek commented 8 years ago

👍

johnynek commented 8 years ago

@jnievelt what is the story here? Should we not merge?

jnievelt commented 8 years ago

I'll squash this and merge today.