[Avro] can't serialize POJO with extra field (that isn't in the schema)

[leonbur]

My POJO is autogenerated from an avro schema file. There are extra (private) fields in the POJO as a result of the generation. The mapper doesn't allow me to serialize this POJO with that very schema even though it (the POJO) contains all the necessary fields.

Example: POJO:

class Person {
        private String name;
        private String unnecessaryField;

       // getters, setters and constructors
    }

Schema:

{
  "name": "Person",
  "type": "record",
  "fields": [
        {
            "name": "name",
                        "type": "string"
                }
    ]
}

Code:

Person person = new Person("Jackie", "bla bla");

AvroSchema s = mapper.schemaFrom(personSchemaStr);
byte[] serialized = mapper.writer(s).writeValueAsBytes(person);

Exception: com.fasterxml.jackson.databind.JsonMappingException: No field named 'unnecessaryField'

[cowtowncoder]

Schema is only applied at encoding/decoding level and format-agnostic databinding unfortunately has no real access to it (at least at this point). You will need to mark unnecessaryField with @JsonIgnore (or use @JsonIgnoreProperties with that name included).

Other format backends do, however, have features to automatically ignore write requests for unknown properties so it might be possible to improve this for Avro as well -- I can't say how easy that would be, but worth checking I think. So I will consider this an RFE for adding such AvroGenerator.Feature.

[santoo1116]

@cowtowncoder could you let us know the status on this? Is this started or in priority? Backward compatibility is a major option for our implementation.

[cowtowncoder]

@santoo1116 If there was something to update, it'd be here. So no progress here. I am not working on this, and I am not aware of anyone else working on this -- and once again, if someone was, I'd expect them to add a note.

So in all likelihood, someone would need to pick this up to help.

[cowtowncoder]

I will add, though, that use of "Config Overrides" from jackson-databind would probably allow specifying equivalent of @JsonIgnoreProperties to use for specific type. Like this (from test src/test/java/com/fasterxml/jackson/databind/deser/filter/IgnorePropertyOnDeserTest.java):

        ObjectMapper mapper = JsonMapper.builder()
            .withConfigOverride(MyPojo.class,
                cfg -> cfg.setIgnorals(JsonIgnoreProperties.Value.forIgnoredProperties("y")))
            .build();

[santoo1116]

@cowtowncoder - Thanks! but every time when there's a new attribute - we have to come back and build our code .

Customized the Avro factory and its working! Is this ok?

AvroFactory avroFactory = new AvroFactory();
avroFactory.configure(Feature.IGNORE_UNKNOWN, true);
ObjectMapper mapper = AvroMapper.builder(avroFactory).build();

[cowtowncoder]

@santoo1116 If that seems to work, sure. It may hide some problems (as it ignores everything), but definitely avoids the problem you have.

[santoo1116]

@cowtowncoder got it, this just ignore the extra properties in our schema - so apart from being backward compatible - do you see any major issues?

[cowtowncoder]

@santoo1116 No, I see it as similar to @JsonIgnoreProperties(ignoreUnknown=true) wrt trade-offs. Nothing technical, just possibility of not catching structural mismatches (like nothing matching).

[santoo1116]

Got it, thanks @cowtowncoder