twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.5k stars 706 forks source link

introduce modes for required ordered serialization #1757

Closed fwbrasil closed 6 years ago

fwbrasil commented 6 years ago

Problem

We're planning to roll out OrderedSerialization to more jobs. The initial plan was to add the RequiredBinaryComparators traits to our execution and job classes, but jobs could fail at runtime if TypedPipe transformations are done outside the job/execution body since the implicit macro wouldn't be available. For instance, we recommend people to extract transformations in companion objects for testability.

Solution

Introduce a mechanism that allows us to add the traits without failing at runtime if a transformation doesn't have the ordered serialization. By overriding requireOrderedSerializationMode, we can set the mode to Log, which will produce an error message at runtime.

Notes

fwbrasil commented 6 years ago

cc/ @johnynek

fwbrasil commented 6 years ago

@johnynek I've just addressed your comments. Thank you for the quick review!

johnynek commented 6 years ago

👍