search

danzafar / tidyspark

tidyspark: a tidyverse implementation of SparkR built for simplicity, elegance, and ease of use.
Other
22 stars 0 forks source link

sparkContext API POC #49

Closed danzafar closed 4 years ago

danzafar commented 4 years ago

@estern95 Here is a PR where we can discuss the sparkContext R4 implementation. I've submitted my POC test. If you pull this branch you should be able to run:

sc <- sparkContext$new()
sc$sparkConf$get(<some_conf>)

and

sc$parallelize(list(1:10), 2)

I'm thinking we do this first, them maybe RDD API if we still like it :)