twitter / scalding

A Scala API for Cascading
http://twitter.com/scalding
Apache License 2.0
3.49k stars 703 forks source link

Simplify the Memory platform using Dagon memoization #1836

Closed johnynek closed 6 years ago

johnynek commented 6 years ago

Using Dagon's Memoize.functionK, it is much easier to write recursive memoized functions without threading through the state yourself.

I changed the code here to do that since we will likely want to copy this approach in the spark backend (which I started to do in #1832 ).

johnynek commented 6 years ago

@non @erik-stripe can you review? you have the most dagon familiarity.

cc @ianoc @fwbrasil

ianoc commented 6 years ago

Erik's review would be better, but all looks pretty sane to me

non commented 6 years ago

LGTM 👍

Not having to explicitly thread the memoization through really improves readability!