Version 6.4.0 failing for Spark 3.5.0

[ravillahari]

Abris Version: 6.4.0 Spark Version: 3.5.0 Databricks Runtime: 14.3

Failing with error: Cause: java.lang.NoClassDefFoundError: org/apache/spark/sql/internal/SQLConf$LegacyBehaviorPolicy$ at org.apache.spark.sql.avro.AvroDeserializer.(AvroDeserializer.scala:62) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.spark.sql.avro.AbrisAvroDeserializer$$anonfun$1.applyOrElse(AbrisAvroDeserializer.scala:39) at org.apache.spark.sql.avro.AbrisAvroDeserializer$$anonfun$1.applyOrElse(AbrisAvroDeserializer.scala:37) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38) at scala.util.Failure.recover(Try.scala:234) at org.apache.spark.sql.avro.AbrisAvroDeserializer.(AbrisAvroDeserializer.scala:37)

In Spark 3.4.0 "LegacyBehaviorPolicy" is available in same file as here: https://github.com/apache/spark/blob/branch-3.4/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

However in Spark 3.5.0 "LegacyBehaviorPolicy" is available and used in package: https://github.com/apache/spark/blob/branch-3.5/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala But is available new package: https://github.com/apache/spark/blob/branch-3.5/sql/api/src/main/scala/org/apache/spark/sql/internal/LegacyBehaviorPolicy.scala

Could this be issue for the above error?

[cerveada]

Yes, it used to be inside SQLConf object, now it is a top level class, so the full class name is different.

[SlevinBE]

A workaround for this issue is available in https://github.com/AbsaOSS/ABRiS/issues/355 However, Abris should probably provide the right version of the spark-avro dependency.