search

JahstreetOrg / spark-on-kubernetes-helm

Spark on Kubernetes infrastructure Helm charts repo
Apache License 2.0
199 stars 75 forks source link

Run livy with customize docker image got the "Read-only file system" error #69

Closed hth1995 closed 3 years ago

hth1995 commented 3 years ago

Hi, I tried to customize the spark image for Livy so that I clone the https://github.com/JahstreetOrg/spark-on-kubernetes-docker repo and build the images by order.

Spark Livy-Spark Livy (I'm use Livy builder from sasnouskikh/livy-builder:0.3)

I can able to run the helm install with the customize livy image but when I create a batch, I got the Read-only file system error. /opt/entrypoint.sh: line 45: /opt/spark/conf/spark-defaults.conf: Read-only file system image

I tried to add the chmod cmd, but It's not working. RUN ["chmod", "777", "-R", "/opt/spark/conf/spark-defaults.conf"]

hth1995 commented 3 years ago

I want to run the python file with a batch session. And the python file needs some extra lib, I tried to add the lib via API but it is not working. So that I tried to customize the Spark image.

Is Livy support adding the python lib via API?