search

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
https://spark.apache.org/
Apache License 2.0
612 stars 118 forks source link

Added configuration properties to inject arbitrary secrets into the driver/executors #479

Closed liyinan926 closed 7 years ago

liyinan926 commented 7 years ago

What changes were proposed in this pull request?

This PR added two classes of configuration properties, namely spark.kubernetes.driver.secrets.[SecretName] and spark.kubernetes.driver.secrets.[SecretName], respectively, for mounting arbitrary secrets onto user-specified paths into the driver and executor Pods. Both properties are for mounting the secret named SecretName onto the path specified by the property value. This PR addresses #397. We were told that PodPreset had been moved out of core entirely and initializers would not go beta until in 1.9. Given that both of them won't make it to beta anytime soon, we need a solution that can unblock us for some use cases on accessing GCP products on GKE.

How was this patch tested?

Unit tests. Manual tests using minikube as the following output driver Pod manifest shows.

spec:
  containers:
    volumeMounts:
    - mountPath: /opt/spark/secrets
      name: spark-gcs-service-account-volume
  volumes:
  - name: spark-gcs-service-account-volume
    secret:
      defaultMode: 420
      secretName: spark-gcs-service-account
mccheah commented 7 years ago

Ah and the merge conflicts. We moved executor pod construction to a separate class so the code in the scheduler backend has to move accordingly.

liyinan926 commented 7 years ago

Addressed comments, merged and resolved conflicts.