search

JahstreetOrg / spark-on-kubernetes-helm

Spark on Kubernetes infrastructure Helm charts repo
Apache License 2.0
198 stars 76 forks source link

Question: Livy Scala Api #70

Open jdcohen220 opened 2 years ago

jdcohen220 commented 2 years ago

What version of scala/spark do client applications need to use? The image says Spark 3.0.1 but Livy api requires Scala 2.11 which only works with earlier versions of Spark.

I am getting a ClassNotFound error when trying to submit a job: 00:20 WARN: [kryo] Unable to load class org.apache.livy.scalaapi.LivyScalaClient$$anon$1 with kryo's ClassLoader. Retrying with current.. 21/08/23 19:14:12 INFO JobWrapper: Failed to run job b3353a3c-2140-4d4c-94c2-520473152ec2 org.apache.livy.shaded.kryo.kryo.KryoException: Unable to find class: org.apache.livy.scalaapi.LivyScalaClient$$anon$1

jahstreet commented 1 year ago

Hi @jdcohen220 , sorry for the late reply, getting back to this project only now 🙏 .

Is it still an issue for you? Happy to guide you through the resolution in case you provide me the details on reproducing it.