Closed balbarka closed 1 month ago
@cwoodcox can you confirm your exception DBR and try again if it wasn't in 10.4 LTS?
I'd like to isolate if this is spark version related.
I created another cluster with 10.4 LTS and Spark 3.2.1, and the same notebook worked swimmingly.
I'm not experienced in Scala, but digging through the source of the API it's complaining about and it doesn't seem to have changed between 3.2.1 and 3.3.0, so it's puzzling.
Adding this to make jar available for Spark 3.3.0 compatibility
Came here just now to report this… unsure if it's the same error as @balbarka but I'm running those same versions on Azure Databricks.
Here's my pyspark:
and my stack trace: