Open lukleh opened 5 years ago
Nomad 0.9.1 Pyspark 2.4.3
0.9.1
2.4.3
example pyspark-shell command running against Nomad cluster:
--conf spark.nomad.sparkDistribution=local:/usr/lib/spark \ --conf spark.dynamicAllocation.enabled=true \ --conf spark.shuffle.service.enabled=true \ --conf spark.dynamicAllocation.minExecutors=1
In pyspark-shell load the following:
When NOT setting spark.dynamicAllocation.maxExecutors and cluster exhausts resources, executors are still queued although all calculation finished. 
spark.dynamicAllocation.maxExecutors
Nomad
0.9.1
Pyspark
2.4.3
example pyspark-shell command running against Nomad cluster:
In pyspark-shell load the following:
When NOT setting
spark.dynamicAllocation.maxExecutors
and cluster exhausts resources, executors are still queued although all calculation finished.