search

kestra-io / plugin-serdes

https://kestra.io/plugins/plugin-serdes/
Apache License 2.0
2 stars 5 forks source link

Fix/allow default value #3

Open corentinghigny opened 4 years ago

corentinghigny commented 4 years ago

I don't even remember why, but while working on avro compatibility check, we found out strange behaviour by the parser if it encounter null value while parsing String. While trying to fix that, a lot of stuff popped out.

We used that opportunity to fix handling of "null" and implement "default" field value (avro spec). We also added a parameter "missingDefaultIsNull", that put "null" as "default" if no default is given

More info on why this is needed on https://issues.apache.org/jira/browse/AVRO-1803 (no "null" default means that if key isn't found in the record, the record is invalid)

This parameter is "true" by default for simplicity and avoid breaking compatibility

It SHOULD not break anything, UT still works, but it might, has before, in the fromMap method of AvroConverter, the only line was

record.put(field.name(), convert(field.schema(), data.get(field.name())));

this didn't check if the key existed in the record, Map().get() returns null if the key doesn't exist

corentinghigny commented 4 years ago

This still needs works : a lot of TODO and FIXME were put in nearly every avro type test I'd like some help on those because I have deadlines coming around :sweat:

corentinghigny commented 4 years ago

Also, missingDefaultIsNull is a lame variable name, some ideas for a better one ?