[bitnami/spark] Executors run in separate pods instead of the worker pods

[ricsi046]

Name and Version

bitnami/spark >= 8.0.3

What architecture are you using?

amd64

What steps will reproduce the bug?

I'm running this on Windows 10 via docker-desktop's Kubernetes. Docker-desktop version is 4.34.0.

1. kubectl create serviceaccount spark-test-acc
2. kubectl create rolebinding spark-test-acc-cluster-admin --clusterrole=cluster-admin --serviceaccount=default:spark-test-acc --namespace=default
3. helm install my-release oci://registry-1.docker.io/bitnamicharts/spark --version 8.0.3 --set serviceAccount.create=false,serviceAccount.name=spark-test-acc (version can be anything >= 8.0.3)
4. kubectl exec -ti --namespace default my-release-spark-worker-0 -- /opt/bitnami/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-test-acc --conf spark.kubernetes.container.image=bitnami/spark:3.5.0-debian-11-r12 --master k8s://https://kubernetes.docker.internal:6443 --conf spark.kubernetes.driverEnv.SPARK_MASTER_URL=spark://my-release-spark-master-svc:7077 --deploy-mode cluster local:///opt/bitnami/spark/examples/jars/spark-examples_2.12-3.5.0.jar 10

Are you using any custom parameters or values?

See step 3.

What is the expected behavior?

When you run the spark-submit command, the executors should run on the workers.

This behavior can be achieved by using older chart versions (<= 8.0.2).

But it can also be achieved by running this command (note how I changed the container image from r12 to r10) kubectl exec -ti --namespace default my-release-spark-worker-0 -- /opt/bitnami/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-test-acc --conf spark.kubernetes.container.image=bitnami/spark:3.5.0-debian-11-r10 --master k8s://https://kubernetes.docker.internal:6443 --conf spark.kubernetes.driverEnv.SPARK_MASTER_URL=spark://my-release-spark-master-svc:7077 --deploy-mode cluster local:///opt/bitnami/spark/examples/jars/spark-examples_2.12-3.5.0.jar 10

(This image change trick doesn't work with the recent chart version though)

What do you see instead?

When you run the spark-submit command, you can see that separate executor pods get created instead of running them on the workers created by the spark chart. These separate pods are not configured properly like the worker pods, so they can't run the program successfully. And I'm sure that this is not the intended behavior either.

[juan131]

Hi @ricsi046

Could you try running the commands below?

export EXAMPLE_JAR=$(kubectl exec -ti --namespace test my-release-spark-worker-0 -- find examples/jars/ -name 'spark-example*\.jar' | tr -d '\r')
kubectl exec -ti --namespace test my-release-spark-worker-0 -- spark-submit --master spark://my-release-spark-master-svc:7077 --class org.apache.spark.examples.SparkPi  --conf spark.standalone.submit.waitAppCompletion=true --deploy-mode cluster $EXAMPLE_JAR 5

[ricsi046]

Hello

Yeah that works. However, in that case it runs in standalone mode.

I'm a bit confused, because the chart seems like it was made for creating a standalone spark cluster, but the readme suggests submitting applications to kubernetes (master is k8s). So can you confirm what is the intended use-case?

I assume standalone mode should be used with this chart since one feature of running spark on kubernetes is that pods only run while the application runs, then they are shut down, which is not true for this chart since the master and worker pods are always running.

[juan131]

Hi @ricsi046

Yes, this chart deploys Spark in a standalone cluster mode.

[ricsi046]

Thanks, in that case I'm closing the issue since not k8s but the spark master created by the chart should be specified as master when running spark-submit.