In #137, a bug was fixed, to preserve the non-nullability of fields from an avro schema to a catalyst schema. I.e. fields that were previously (wrongly) nullable in the output schema are now non-nullable. However, this creates an incompatible change if the output is a kafka topic with backwards schema compatibility.
Therefore, a flag should be added to the ConfluentAvroDecodingTransformer that enables the old (wrong) behaviour for backwards compatibility for use cases where nullability preservation is not important.
In #137, a bug was fixed, to preserve the non-nullability of fields from an avro schema to a catalyst schema. I.e. fields that were previously (wrongly) nullable in the output schema are now non-nullable. However, this creates an incompatible change if the output is a kafka topic with backwards schema compatibility.
Therefore, a flag should be added to the
ConfluentAvroDecodingTransformerthat enables the old (wrong) behaviour for backwards compatibility for use cases where nullability preservation is not important.