Executors do not scale down in dynamic allocation

[lukleh]

Nomad 0.9.1
Pyspark 2.4.3

example pyspark-shell command running against Nomad cluster:

    --conf spark.nomad.sparkDistribution=local:/usr/lib/spark \
    --conf spark.dynamicAllocation.enabled=true \
    --conf spark.shuffle.service.enabled=true \
    --conf spark.dynamicAllocation.minExecutors=1 \
    --conf spark.dynamicAllocation.maxExecutors=30

In pyspark-shell load the following:

• generate sample data https://gist.github.com/lukleh/13456e9a31def32f1b1eb07d659347e1
• force computation https://gist.github.com/lukleh/dc861417f89b2255bf97be8606f2a430

After spark.dynamicAllocation.executorIdleTimeout executors do not get killed and the following logs appear instead:

19/07/28 13:01:05 WARN NomadClusterSchedulerBackend: Ignoring request to kill 15 executor(s): {{EXECUTORS IDS SKIPPED}} (not yet implemented for Nomad)  
19/07/28 13:01:05 WARN ExecutorAllocationManager: Unable to reach the cluster manager to kill executor/s {{EXECUTORS IDS SKIPPED}} or no executor eligible to kill!

which is poiting to https://github.com/hashicorp/nomad-spark/blob/nomad-spark-2.4.3/resource-managers/nomad/src/main/scala/org/apache/spark/scheduler/cluster/nomad/NomadClusterSchedulerBackend.scala#L188 missing implementation

similar issue https://github.com/hashicorp/nomad-spark/issues/20

[cgbaker]

Hi @lukleh , the Nomad Spark integration does not implement downscaling when using dynamic executors. Upcoming features road-mapped for Nomad 0.10.x will allow us to decouple the shuffle service processes from the executor processes and support proper downscaling. I'll leave this issue open to track that.