zbenta
commented
2 years ago Hi @jdries, just tried to follow your tutorial and found that I'm unable to setup the spark-operator in the current version(v1.16.15) of kubernetes we are using in the cluster. I've tried deploying it on another cluster with a newer version(v1.21.6) and it worked, now, we are having issues with the spark-jobs. When we try to deploy it as follows:
[root@k8s-test-cluster-k8s-master-nf-1 ~]# helm install myspark vito/sparkapplication --version 0.5.0 --create-namespace --namespace spark-jobs -f values.yaml
W0607 11:00:37.367991 11917 warnings.go:70] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W0607 11:00:37.369756 11917 warnings.go:70] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W0607 11:00:38.484370 11917 warnings.go:70] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W0607 11:00:38.484556 11917 warnings.go:70] networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME: myspark
LAST DEPLOYED: Tue Jun 7 11:00:36 2022
NAMESPACE: spark-jobs
STATUS: deployed
REVISION: 1
TEST SUITE: None
It states that it is ok, but if we consult the logs from the pod that is started we see the following:
spark-kubernetes-driver
++ id -u
spark-kubernetes-driver
+ myuid=18585
spark-kubernetes-driver
++ id -g
spark-kubernetes-driver
+ mygid=18585
spark-kubernetes-driver
+ set +e
spark-kubernetes-driver
++ getent passwd 18585
spark-kubernetes-driver
+ uidentry=spark:x:18585:18585::/opt/spark/work-dir:/bin/bash
spark-kubernetes-driver
+ set -e
spark-kubernetes-driver
+ '[' -z spark:x:18585:18585::/opt/spark/work-dir:/bin/bash ']'
spark-kubernetes-driver
+ SPARK_CLASSPATH=':/usr/local/spark/jars/*'
spark-kubernetes-driver
+ env
spark-kubernetes-driver
+ grep SPARK_JAVA_OPT_
spark-kubernetes-driver
+ sort -t_ -k4 -n
spark-kubernetes-driver
+ sed 's/[^=]*=\(.*\)/\1/g'
spark-kubernetes-driver
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
spark-kubernetes-driver
+ '[' -n '' ']'
spark-kubernetes-driver
+ '[' '' == 3 ']'
spark-kubernetes-driver
+ '[' -n /usr/hdp/current/hadoop-client ']'
spark-kubernetes-driver
+ '[' -z '' ']'
spark-kubernetes-driver
++ /usr/hdp/current/hadoop-client/bin/hadoop classpath
spark-kubernetes-driver
/opt/entrypoint.sh: line 57: /usr/hdp/current/hadoop-client/bin/hadoop: No such file or directory
spark-kubernetes-driver
+ export SPARK_DIST_CLASSPATH=
spark-kubernetes-driver
+ SPARK_DIST_CLASSPATH=
spark-kubernetes-driver
+ '[' -z x ']'
spark-kubernetes-driver
+ SPARK_CLASSPATH='/etc/hadoop/conf::/usr/local/spark/jars/*'
spark-kubernetes-driver
+ echo 'My start command is driver'
spark-kubernetes-driver
My start command is driver
spark-kubernetes-driver
+ case "$1" in
spark-kubernetes-driver
+ shift 1
spark-kubernetes-driver
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@")
spark-kubernetes-driver
+ exec /usr/bin/tini -s -- /usr/local/spark/bin/spark-submit --conf spark.driver.bindAddress=10.233.87.198 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.deploy.PythonRunner local:///usr/local/lib/python3.7/dist-packages/openeogeotrellis/deploy/kube.py
spark-kubernetes-driver
WARNING: An illegal reflective access operation has occurred
spark-kubernetes-driver
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/spark-3.2.0/jars/spark-unsafe_2.12-3.2.0.jar) to constructor java.nio.DirectByteBuffer(long,int)
spark-kubernetes-driver
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
spark-kubernetes-driver
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
spark-kubernetes-driver
WARNING: All illegal access operations will be denied in a future release
spark-kubernetes-driver
22/06/07 11:33:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
spark-kubernetes-driver
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
spark-kubernetes-driver
22/06/07 11:33:21 WARN DependencyUtils: Local jar /opt/geotrellis-extensions-2.2.0-SNAPSHOT.jar does not exist, skipping.
spark-kubernetes-driver
22/06/07 11:33:21 WARN DependencyUtils: Local jar /opt/geotrellis-backend-assembly-0.4.6-openeo.jar does not exist, skipping.
spark-kubernetes-driver
python3: can't open file '/usr/local/lib/python3.7/dist-packages/openeogeotrellis/deploy/kube.py': [Errno 2] No such file or directory
spark-kubernetes-driver
22/06/07 11:33:21 INFO ShutdownHookManager: Shutdown hook called
spark-kubernetes-driver
22/06/07 11:33:21 INFO ShutdownHookManager: Deleting directory /tmp/spark-9ebcd8be-dc28-4cbd-9c56-e68a90bd6050It looks like the image is missing a few dependecies.
In the meanwhile we will try to upgrade our kubernetes cluster and try to redeploy the service.
tcassaert
Jaapel
The endpoint is deployed but with all these errors. We have no idea if it is working properly, we believe that running the notebook would confirm that all is as it should be. Is there any other way to confirm if all is setup correctly on our endpoint?
jdries
We get the output from the myspark-driver pod:
Thanks @tcassaert we'll change it to the default port and redeploy zookeeper to see if that is the issue.
enolfc
Upgrade openEO to latest version. @zbenta We updated our documentation to reflect latest upgrades to software stack: https://github.com/Open-EO/openeo-geotrellis-kubernetes/commit/a8147962d49555d366b00f64f7b27ff62c5f712e
Can you try again based on those instructions, and report issues here? @tcassaert is available to follow up more quickly (than I can)!