Basic Secure HDFS Support [514]

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/

https://spark.apache.org/

Apache License 2.0

612 stars 118 forks source link

Basic Secure HDFS Support [514] #540

Closed ifilonenko closed 6 years ago

ifilonenko commented 6 years ago

What changes were proposed in this pull request?

This is the on-going work of setting up Secure HDFS interaction with Spark-on-K8S #514 The architecture is discussed in this community-wide google doc This initiative can be broken down into 3 stages.

STAGE 1

[x] Detecting HADOOP_CONF_DIR environmental variable and using Config Maps to store all Hadoop config files locally, while also setting HADOOP_CONF_DIR locally in the driver / executors

STAGE 2

[x] Grabbing TGT from LTC or using keytabs+principle and creating a DT that will be mounted as a secret

STAGE 3

[x] Driver + Executor Logic

mccheah commented 6 years ago

No unit tests exist for the new classes - were those drafted up in the original as well? Can we include them here? Some of the utility classes created for testing lack that context.

ifilonenko commented 6 years ago

Unit tests were a massive portion of the PR that would almost double line count. Should I include them in this or separate PR?

ifilonenko commented 6 years ago

rerun integration tests please

ifilonenko commented 6 years ago

rerun integration tests please

ifilonenko commented 6 years ago

rerun integration tests please

ifilonenko commented 6 years ago

Rerun integration tests please

ifilonenko commented 6 years ago

rerun integration tests please

ifilonenko commented 6 years ago

Would like to merge since I have already recieved two LGTM. Any contentions / comments?

liyinan926 commented 6 years ago

Any more comments? If not, will merge by EOD today.