search

AbsaOSS / ABRiS

Avro SerDe for Apache Spark structured APIs.
Apache License 2.0
230 stars 75 forks source link

Is Abris supposed to work with Spark 3.1.1? #221

Closed hatharom closed 3 years ago

hatharom commented 3 years ago

Description mentions only 3.0.x with avro v1.8.0 . Spark 3.1.1 brings in v1.9.2 (tried also with explicit 1.8.0 declaration and also with Abris 4.2.0/3.1.x), but can't get it to work.

I don't include my logs right now, it is just a general question : is Spark 3.1.x supported?

kevinwallimann commented 3 years ago

Hi @hatharom I have used Abris v4.2.0 with Spark 3.1.1, although not all features. Looking at their pom file, I still see Spark 3.1.1 uses avro v1.8.0. Where did you see it uses avro v1.9.2?

hatharom commented 3 years ago

You right, I double checked , it is not 1.9.x, it is 1.8.0. My mistake.