In Spark 3.2.1, a typo in the method signature for DataSourceUtils.creteDateRebaseFuncInWrite was fixed. https://github.com/apache/spark/pull/34978 Of course, this change is neither backward nor forward compatible. Since the spark-avro jar is a compile-time dependency for ABRiS, other applications that create a fat-jar with the spark-avro v3.2.0 will experience a NoSuchMethodError when running this against Spark 3.2.1 (see below), because the spark-avro jar is usually not part of the Spark distribution.
Cause: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.DataSourceUtils$.creteDateRebaseFuncInRead(Lscala/Enumeration$Value;Ljava/lang/String;)Lscala/Function1;
at org.apache.spark.sql.avro.AvroDeserializer.<init>(AvroDeserializer.scala:67)
at org.apache.spark.sql.avro.AvroDeserializer.<init>(AvroDeserializer.scala:61)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.sql.avro.AbrisAvroDeserializer$$anonfun$1.applyOrElse(AbrisAvroDeserializer.scala:40)
at org.apache.spark.sql.avro.AbrisAvroDeserializer$$anonfun$1.applyOrElse(AbrisAvroDeserializer.scala:38)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
at scala.util.Failure.recover(Try.scala:234)
In Spark 3.2.1, a typo in the method signature for
DataSourceUtils.creteDateRebaseFuncInWrite
was fixed. https://github.com/apache/spark/pull/34978 Of course, this change is neither backward nor forward compatible. Since the spark-avro jar is a compile-time dependency for ABRiS, other applications that create a fat-jar with the spark-avro v3.2.0 will experience aNoSuchMethodError
when running this against Spark 3.2.1 (see below), because the spark-avro jar is usually not part of the Spark distribution.