search

Ten0 / serde_avro_fast

An idiomatic implementation of serde/avro (de)serialization
GNU Lesser General Public License v3.0
20 stars 4 forks source link

Allow setting default value for RecordField #24

Open alexeiverbny opened 2 weeks ago

alexeiverbny commented 2 weeks ago

Hello,

I am wondering if it would be possible to allow the user to set or change a default field on the RecordField struct. This is possible in apache_avro: https://docs.rs/apache-avro/latest/apache_avro/schema/struct.RecordField.html.

The reason I am asking for this is that is required if you want to properly handle schema evolution while using trino to read avro data: https://trino.io/docs/current/connector/hive.html#avro-schema-evolution. In particular, we are writing structs to tables that are read by trino. If we want to add a field to that struct, the schema must have a default value in order for trino to be able to read older data that was written before the field was added.

Please let me know if what I am asking for is unclear.

Thank you, Alex

Ten0 commented 3 days ago

Hello! Thanks for opening this issue. I didn't answer right away because I wanted to mediate a bit on it, but it's been a while and I haven't come to it yet so I'm just going to lay my thoughts here.

I understand the use-case and I think it makes sense to have it. However I'd also like the existence of this field to not be misleading API-wise on that it's actually used for anything within serde_avro_fast, so I have to meditate a bit on what that would look like. (Maybe something generic that's essentially flattened like extra_attributes: Vec<(&'static str, serde_json::Value)> on the relevant schema types, which would have the advantage of also being very flexible...)

alexeiverbny commented 2 days ago

Thanks for the response. The generic option you suggested seems reasonable to me.

Ten0 commented 2 days ago

The downside of that though is that if we then add a field to the structured list it won't show in extra_attributes, which would then be a breaking change...