search

palantir / spark

Palantir Distribution of Apache Spark
Apache License 2.0
67 stars 51 forks source link

Upgrade Avro to 1.10.2 #743

Closed rshkv closed 3 years ago

rshkv commented 3 years ago

This a backport to our Spark 2 branch. Upgrade to Avro 1.10.2.

1.10.2 comes with some fixes we're interested in, particularly AVRO-2944 to fix some failures we're seeing (ticket).

Beside the upgrade, you'll see we're dropping the avro.mapred.classifier. It was necessary previously to pull an Avro version compatible with Hadoop 2 as opposed to Hadoop 1. As of 1.9.0 Hadoop 2 is the default and the classifier no longer works (AVRO-2059).

LorenzoMartini commented 3 years ago

Solved with https://github.com/palantir/spark/pull/758