search

confluentinc / schema-registry

Confluent Schema Registry for Kafka
https://docs.confluent.io/current/schema-registry/docs/index.html
Other
2.22k stars 1.11k forks source link

Serialisation of JSON with schema fails when additionalProperties is set to false. #2015

Open schubertf opened 3 years ago

schubertf commented 3 years ago

Serialisation of JSON with schema fails when additionalProperties is set to false.

We have a data model that requires additionalProperties to be set to either true or false at various points in the hierarchy, i.e. some beans will have additionalProperties set to true and others will have it set to false. We are using com.kjetland.jackson.jsonSchema.annotations.JsonSchemaInject to achieve this.

On doing so the schema seems to be generated correctly with additionalProperties set to either true or false as required, however when producing a message (with the configuration property json.oneof.for.nullables set to the default of true) the serialisation fails with an error similar to the following...

Exception in thread "main" org.apache.kafka.common.errors.SerializationException: Error serializing JSON message
Caused by: org.apache.kafka.common.errors.SerializationException: JSON explore.Outer@53499d85 does not match schema {"$schema":"http://json-schema.org/draft/2019-09/schema#","title":"Outer","type":"object","additionalProperties":true,"properties":{"inner":{"oneOf":[{"type":"null","title":"Not included"},{"$ref":"#/definitions/Inner"}],"description":"Inner property.","additionalProperties":false}},"definitions":{"Inner":{"type":"object","additionalProperties":false,"properties":{"prop1":{"oneOf":[{"type":"null","title":"Not included"},{"type":"string"}],"description":"Property 1."},"prop2":{"oneOf":[{"type":"null","title":"Not included"},{"type":"string"}],"description":"Property 2."}}}}}
Caused by: org.everit.json.schema.ValidationException: #/inner: #: only 1 subschema matches out of 2
    at org.everit.json.schema.ValidationException.copy(ValidationException.java:486)
    at org.everit.json.schema.DefaultValidator.performValidation(Validator.java:67)
    at org.everit.json.schema.Schema.validate(Schema.java:152)
    at io.confluent.kafka.schemaregistry.json.JsonSchema.validate(JsonSchema.java:292)
    at io.confluent.kafka.serializers.json.AbstractKafkaJsonSchemaSerializer.serializeImpl(AbstractKafkaJsonSchemaSerializer.java:102)
    at io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializer.serialize(KafkaJsonSchemaSerializer.java:85)
    at org.apache.kafka.common.serialization.Serializer.serialize(Serializer.java:62)
    at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:925)
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:885)
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:773)
    at explore.Main.send(Main.java:48)
    at explore.Main.main(Main.java:30)

If we override the default and set the configuration property json.oneof.for.nullables to false the message is serialised correctly, albeit with a slightly different schema as expected. See attached Maven project to reproduce. json.schema.oneof.additional.properties.zip The serialisation of the JSON payload should work regardless of the value of the json.oneof.for.nullables configuration property.

rayokota commented 3 years ago

@schubertf , thanks for the reproduction. It seems the JsonSchemaInject of additionalProperties does not play nicely with json.oneof.for.nullables=true due to an issue with the underlying JSON schema library.

I've committed https://github.com/confluentinc/schema-registry/pull/2016, but that only allows you to toggle additionalProperties for all objects, not customize it for each object.

schubertf commented 3 years ago

Thanks for your reply @rayokota. As I mentioned, we need to be able to customise additionalProperties for each object. We can currently do this by setting json.oneof.for.nullables to false, but would like to be able to do this and use oneOf for nullable properties.