search

AbsaOSS / ABRiS

Avro SerDe for Apache Spark structured APIs.
Apache License 2.0
231 stars 76 forks source link

ABRiS Version 3.2.2 with Spark 3.2.1 Throws Error #321

Closed Rap70r closed 1 year ago

Rap70r commented 1 year ago

Hello,

I have a custom application that I'm running on EMR cluster. The application is using ABRiS Version 3.2.2. It works great with Spark version 3.1.2. However, when I use the application with Spark 3.2.1 it throws the following error: 

java.lang.AbstractMethodError: Method za/co/absa/abris/avro/sql/AvroDataToCatalyst.child()Lorg/apache/spark/sql/catalyst/trees/TreeNode; is abstract

Upgrading to latest ABRiS version is not an option as this point. Can you please provide some guidance on how to use ABRiS version 3.2.2 with Spark 3.2.1?

Thank you

cerveada commented 1 year ago

Hello, Spark 3.2.1 is supported by Abris version 6.2.0+

see: https://github.com/AbsaOSS/ABRiS#supported-versions

Upgrading to latest ABRiS version is not an option as this point. Can you please provide some guidance on how to use ABRiS version 3.2.2 with Spark 3.2.1?

Oh ok, then the only alternative to upgrading is to fork Abris and fix the compatibility issues yourself.

Rap70r commented 1 year ago

Thank you, @cerveada,