search

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
https://spark.apache.org/
Apache License 2.0
612 stars 118 forks source link

fix #538 - make /opt/spark/work-dir writable by gid 0 #560

Open erikerlandson opened 6 years ago

erikerlandson commented 6 years ago

What changes were proposed in this pull request?

make /opt/spark/work-dir writable by gid 0 to operate better in anonymous uid scenarios

(Please fill in changes proposed in this fix)

How was this patch tested?

I'm using CI infra to fully test image creation and exercising via the unit/integration tests

erikerlandson commented 6 years ago

Note, altering the permissions of /opt/spark/work-dir appears to be sufficient, not all of $SPARK_HOME as I assumed originally when I wrote up #538

erikerlandson commented 6 years ago

Tangentially, I fixed a missing && after mkdir -p /opt/spark/work-dir - which makes me wonder what that RUN complex was actually doing

erikerlandson commented 6 years ago

@foxish ptal - also curious what you think about the missing && that was there

foxish commented 6 years ago

Do we need this to go into the upstream as well? https://github.com/apache/spark/pull/19717 has the spark-base docker-file within the PR.

foxish commented 6 years ago

I think for people looking to customize the image using ours as base, this might be an issue - to have the owner be root. I had a discussion on the thread with Mridul in https://github.com/apache/spark/pull/19717#discussion_r154834099. Assuming that it'll be root might be tenuous when someone uses PodSecurityPolicy to enforce a certain user.

cc @liyinan926