RevolutionAnalytics / dplyr-spark

spark backend for dplyr
48 stars 18 forks source link

memoization #22

Open piccolbo opened 9 years ago

piccolbo commented 9 years ago

is it possible to recover some of the goodness of memoization in dplyr.spark? The reason is: Big data operations can be costly, in $$ and time. When programming interactively, one may run a program, inspect the results, add another step, inspect the results and so on. The computation from the previous steps should not be repeated, but space should also be used carefully.