search

AbsaOSS / ABRiS

Avro SerDe for Apache Spark structured APIs.
Apache License 2.0
230 stars 75 forks source link

Upgrade to Spark v3.2.1 #280

Closed kevinwallimann closed 2 years ago

kevinwallimann commented 2 years ago

In Spark 3.2.1, a typo in the method signature for DataSourceUtils.creteDateRebaseFuncInWrite was fixed. https://github.com/apache/spark/pull/34978 Of course, this change is neither backward nor forward compatible. Since the spark-avro jar is a compile-time dependency for ABRiS, other applications that create a fat-jar with the spark-avro v3.2.0 will experience a NoSuchMethodError when running this against Spark 3.2.1 (see below), because the spark-avro jar is usually not part of the Spark distribution.

  Cause: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.DataSourceUtils$.creteDateRebaseFuncInRead(Lscala/Enumeration$Value;Ljava/lang/String;)Lscala/Function1;
  at org.apache.spark.sql.avro.AvroDeserializer.<init>(AvroDeserializer.scala:67)
  at org.apache.spark.sql.avro.AvroDeserializer.<init>(AvroDeserializer.scala:61)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at org.apache.spark.sql.avro.AbrisAvroDeserializer$$anonfun$1.applyOrElse(AbrisAvroDeserializer.scala:40)
  at org.apache.spark.sql.avro.AbrisAvroDeserializer$$anonfun$1.applyOrElse(AbrisAvroDeserializer.scala:38)
  at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
  at scala.util.Failure.recover(Try.scala:234)