search

AbsaOSS / ABRiS

Avro SerDe for Apache Spark structured APIs.
Apache License 2.0
230 stars 75 forks source link

Allow customization of avro output #223

Closed edwardcapriolo closed 2 years ago

edwardcapriolo commented 3 years ago

This addresses #220. The largest problem was that the current tools generate a schema that wont compile with: https://avro.apache.org/docs/current/gettingstartedjava.html

The largest problem was package name clash with class names. Solved by appending a namespace in the example. Also convention is records to be upper case. With this code the default behavior is preserved, but users can plug in logic that works for them.

cerveada commented 2 years ago

The other way to solve this issue is to modify the schema after it is created. You can see how it is done here:

https://github.com/AbsaOSS/hyperdrive/blob/v4.5.1/ingestor-default/src/main/scala/za/co/absa/hyperdrive/ingestor/implementation/utils/AbrisConfigUtil.scala#L105

https://github.com/AbsaOSS/hyperdrive/blob/v4.5.1/ingestor-default/src/main/scala/za/co/absa/hyperdrive/ingestor/implementation/transformer/avro/confluent/ConfluentAvroEncodingTransformer.scala#L112

Since this seems to be working well without need to change the Abris converter, I will close this PR.