Open AWebNagra opened 3 months ago
Be sure the spark version is the scala 2.13 version?
I can confirm spark 3.5.1 and almond 0.14.0-RC14 work fine:
Specific setup: https://github.com/coreyoconnor/nix_configs/blob/dev/modules/ufo-k8s/almond-2/Dockerfile
Hello,
We recently tried using almond as a scala kernel within our jupyterlab environment but we are encountering errors when trying to use recent versions of spark.
Spark versions tested : 3.5.0 Scala version : 2.13.8 (scala version used in spark 3.5.0) Java version : 17.0.2 Almond versions (tried) : 0.13.14 and 0.14.0-RC13
The errors arise when trying to send code to the execs, hence inducing serialization and deserialization. Other operations work fine (count, show, etc..)
Here's a minimalistic example causing the error :
and the error is :
Note that running the exact same code in a spark-shell on the jupyterlab instance works fine, hence the problem must come from almond. Our best guess is that the classpath used by almond imports mismatched versions of some libs, but we don't have any proof that this could be the issue
Second note : we tried using our own spark installed in the jupyterlab image AND installing spark directly with ivy from a scala notebook, both induce the same error.
Does anyone have any idea what could be causing this issue ?