search

databricks / spark-perf

Performance tests for Apache Spark
Apache License 2.0
379 stars 203 forks source link

How many executors, cores per executor #98

Closed SiGe closed 8 years ago

SiGe commented 8 years ago

Hi,

I am having trouble figuring out the number of executors that I need to set. I am running the glm-classification on Spark 1.5/Yarn/CDH-5.5, and it is only using one executor (and I believe one thread).

I've tried setting spark.executor.instances and spark.executor.cores to 6/8 respectively (using a cluster of 6 machines each with 8 cores). But the classificaiton is still only using one executor and one core.

Any thoughts are appreciated!

SiGe commented 8 years ago

Fixed ... num_partitions had scaling enabled ... and I was using a very smaller scale, which led to one partition.