AbsaOSS / ABRiS

Avro SerDe for Apache Spark structured APIs.
Apache License 2.0
227 stars 73 forks source link

Support upcoming constructor change in Spark 3.5.1-SNAPSHOT #353

Closed kevinwallimann closed 7 months ago

kevinwallimann commented 7 months ago

Closes #352

Support for the “enableStableIdentifiersForUnionType” option introduced in https://github.com/apache/spark/pull/41263/ was not added. Instead, the option is set to false, which is the default value as documented in https://spark.apache.org/docs/latest/sql-data-sources-avro.html#data-source-option. Users can still use the feature by providing their own SchemaConverter implementation, as explained in https://github.com/absaoss/abris?tab=readme-ov-file#custom-data-conversions If there is demand to directly support this feature through Abris, it can be implemented, but it would require another reflection-based differentiation among Spark versions that support it or not.

github-actions[bot] commented 7 months ago

JaCoCo code coverage report

File Coverage [60.54%] :x:
AbrisAvroDeserializer.scala 60.54% :x:
Total Project Coverage 81.69% :green_apple: