Open networkingana opened 1 month ago
Is there any specific reason for setting the working directory to /opt/spark-jars/
and at the same time using the same folder to keep your .jar files? Try changing either the working directory or the folder to keep the .jar files under.
@networkingana Have you tried running the app with user 185 and group 185, as the spark user's uid and gid are 185.
$ kubectl run --rm --restart=Never -i -t spark --image=spark:3.5.0 -- bash
If you don't see a command prompt, try pressing enter.
spark@spark:/opt/spark/work-dir$ id
uid=185(spark) gid=185(spark) groups=185(spark)
I resolved this issue with changing my Dockerfile, correcting the permissions and ownership of my files. One strange thing is that my application doesn't print its logs
spark.driver.extraClassPath: local:///opt/spark-jars/*
spark.driver.extraJavaOptions: -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Duser.timezone=UTC
-Dlog4j2.configurationFile=file:///opt/spark-jars/log4j2.xml --add-opens java.base/java.nio=ALL-UNNAMED --add-opens
java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/jdk.internal.misc=ALL-UNNAMED --add-opens
java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/sun.security.ssl=ALL-UNNAMED --add-opens
java.base/java.util=ALL-UNNAMED
spark.executor.extraClassPath: local:///opt/spark-jars/*
spark.executor.extraJavaOptions: -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Dlog4j2.configurationFile=file:///opt/spark-jars/log4j2.xml
i'm using the same log4j2.xml file as with older spark versions
@networkingana How did you actually fix the problem? "changing my Dockerfile, correcting the permissions and ownership of my files" is too vague. I am encountering the same issue and I am trying to find a solution.
Same issue here, do you have any updates on it ?
I ended up writing my dockerfile like this
FROM spark:3.5.0
WORKDIR /opt/spark-jars/
ADD build/libs/DataAnalytics*.jar ./DataAnalytics.jar
ADD log4j-core-2.19.0.jar ./log4j-core-2.19.0.jar
ADD log4j-api-2.19.0.jar ./log4j-api-2.19.0.jar
ADD src/main/resources/log4j2.xml ./log4j2.xml
ADD spark-3-rules.yaml ./spark-3-rules.yaml
ADD jmx_prometheus_javaagent-0.11.0.jar ./jmx_prometheus_javaagent-0.11.0.jar
# Spark base image uses user and group 185
USER root
RUN chown -R 185:185 /opt/spark-jars && chmod 777 /opt/spark-jars/*
WORKDIR /opt/spark/work-dir
USER spark
I think the key thing was going back to workdir /opt/spark/work-dir and as well owning my jar files by the spark user which is in the spark:3.5.0 base image
I changed the base image from gcr.io spark:3.1.1 to spark:3.5.0 and built my dockerfile from there where it has my DataAnalyticsReporting.jar
This is the error when the spark-operator tries to start my application
` This is my SparkApplication YAML
and this is my Dockerfile
`
I tried to play and change different users, securityContext and podSecurityContext, but with no success, please let me know how should i configure this