search

data-derp / small-exercises

6 stars 9 forks source link

[Content] [Wrangling in Spark] 8 vs 2 partitions - Exercise Wrangling in Spark - Spark overview - 04: SparkContext and Simple RDDs #33

Closed kelseymok closed 1 year ago

kelseymok commented 1 year ago

In cell 13 the comment "# By default distributed in 8 partitions" is no longer accurate. Using the min cluster setting of 2 core per worker, i'll get 2 tasks (partitions) in the spark job.