radanalyticsio / silex

something to help you spark
Apache License 2.0
65 stars 13 forks source link

Implement `scan` and `scanLeft` for Apache Spark RDD #28

Closed erikerlandson closed 8 years ago

erikerlandson commented 9 years ago

My blog posts describing how these work: http://erikerlandson.github.io/blog/2014/08/12/implementing-parallel-prefix-scan-as-a-spark-rdd-transform/ http://erikerlandson.github.io/blog/2014/08/09/implementing-an-rdd-scanleft-transform-with-cascade-rdds/