search

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
https://spark.apache.org/
Apache License 2.0
612 stars 118 forks source link

Set service account for executors #613

Open witten opened 6 years ago

witten commented 6 years ago

To use custom/private Docker images for Spark drivers and executors, one needs to specify an image pull secret. The consensus in https://github.com/apache-spark-on-k8s/spark/issues/42 was to set the image pull secret onto the default service account in the namespace, and implicitly use that. However, in the case of a cluster that's using RBAC as per this Spark on K8s documentation, Spark will be using a non-default service account, and the image pull secret should be on that service account rather than a global namespace service account.

The difficulty is that, as far as I can tell, there's no way to tell Spark on K8s to spin up executors to use that custom service account, even though there is a way to set the driver's service account with spark.kubernetes.authenticate.driver.serviceAccountName. So the ask in this ticket is to add a comparable mechanism for setting the service account to use for the executors, so that the executors can use the same RBAC configuration and image pull secrets as the driver.

liyinan926 commented 6 years ago

So sounds like we should extend the scope of the configuration property to cover both the driver and executors, e.g., something like spark.kubernetes.authenticate.serviceAccountName.

witten commented 6 years ago

That would work great. Or separate settings for each of driver and executor. Either approach would satisfy this use case.

ku-s-h commented 6 years ago

+1