conf parameter spark.kubernetes.authenticate.driver.serviceAccountName doesn't seem to be respected

[luck02]

Situation: I have an AWS hosted k8s test cluster up and running, we're using it for integration tests for our regular environment. It is, AFAIK, fully functional.

I have built a set of executor, driver, and spark-init images.

I can run my job on minikube where I don't need to mess with serviceAccounts.

In my test cluster however I do need to provide credentials or service accounts.

I am using kubectl proxy and my spark-submit looks like this:

SPARK_DRIVER="$ECR_URL/spark-driver-py:latest"
SPARK_EXECUTOR="$ECR_URL/spark-executor-py:latest"
SPARK_INIT="$ECR_URL/spark-init-latest"

  ./infra/spark-2.2.0-k8s-0.3.0-bin-2.7.3/bin/spark-submit \
      --verbose \
      --deploy-mode cluster \
      --master k8s://http://127.0.0.1:8001 \
      --kubernetes-namespace default \
      --jars local:/opt/jars/hadoop-aws-2.7.3.jar,local:/opt/jars/aws-java-sdk-1.7.4.jar,local:/opt/jars/joda-time-2.9.3.jar,local:/opt/jars/mysql-connector-java-5.1.35.jar \
      --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-k8s-service-account \
      --conf spark.kubernetes.docker.image.pullPolicy=Always \
      --conf spark.driver.memory=1G \
      --conf spark.driver.cores=1 \
      --conf spark.executor.instances=1 \
      --conf spark.executor.cores=1 \
      --conf spark.executor.memory=500m \
      --conf spark.app.name=spark-pi \
      --conf spark.kubernetes.driver.docker.image=$SPARK_DRIVER \
      --conf spark.kubernetes.executor.docker.image=$SPARK_EXECUTOR \
      --conf spark.kubernetes.initcontainer.docker.image=$SPARK_INIT \
      --conf spark.kubernetes.driver.annotations="kube2iamrole"="role-name" \
      --conf spark.kubernetes.executor.annotations="kube2iamrole"="role-name" \
      --conf spark.kubernetes.driver.labels=role=podPresetRole \
      --conf spark.kubernetes.executor.labels=role=podPresetRole \
      local:/opt/jobs/etc/ComputeAndLoad__Manifest.py

The pod spec that gets generated shows the following:

serviceAccountName:default
serviceAccount:default

In my pod log I get:

2017-09-16 22:32:10 ERROR KubernetesClusterSchedulerBackend:91 - Executor cannot find driver pod.
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://kubernetes.default.svc/api/v1/namespaces/default/pods/spark-pi-1505601113438-driver. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. User "system:serviceaccount:default:default" cannot get pods in the namespace "default"..

Kubernetes version:

⇒  kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"922a86cfcd65915a9b2f69f3f193b8907d741d9c", GitTreeState:"clean", BuildDate:"2017-07-21T19:06:19Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4+coreos.0", GitCommit:"4bb697e04f7c356347aee6ffaa91640b428976d5", GitTreeState:"clean", BuildDate:"2017-08-22T08:43:47Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

I've seen mention of issues here: https://github.com/apache-spark-on-k8s/spark/issues/448 but that looks like it was merged awhile back.

You can see from my spark-submit call the version i'm on currently of spark-k8s ./infra/spark-2.2.0-k8s-0.3.0-bin-2.7.3

[luck02]

Ok, just found the commit from @kimoonkim and it's not in the release I'm using.

I'll build a new snapshot this evening. Any chance of a new release coming soon?

[luck02]

Ok, that worked!