search

danzafar / tidyspark

tidyspark: a tidyverse implementation of SparkR built for simplicity, elegance, and ease of use.
Other
22 stars 0 forks source link

Remove global sc #58

Closed danzafar closed 4 years ago

danzafar commented 4 years ago

Big update, when I created the SparkContext R6 class i also made it so that spark_session could put sc into the global environment. Big mistake, CRAN does not allow this. With this update I've created a companion SparkSession R6 class and removed the input into the global environment. Now the suggested workflow is:

spark <- spark_session()
sc <- spark$sparkContext

You could also do:

spark_session()
sc <- SparkContext$new()