twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.5k stars 706 forks source link

Add Evidence parameter to identity reduce operations #1783

Closed johnynek closed 6 years ago

johnynek commented 6 years ago

This makes use of our Leibniz-like type EqTypes to remove several casts. This equips operations that are doing identity reduces to prove to scalac that the input value type is the same as the output.

The main value here is just to remove casts in current and future optimization code. Each cast is a danger, even when correct, because it

  1. makes is more tolerant of casts which in other cases may be unsafe
  2. may incorrectly become unsafe due to code change, but not detected by the compiler during refactoring.
johnynek commented 6 years ago

@non @erik-stripe can you take a look at this one?

I think you are very familiar with these Leibniz tricks.

non commented 6 years ago

LGTM 👍 (love that Leibniz!)