Json schema validation fails when adding new properties

[devrios]

Description

The dotnet client uses 'NJsonSchema' to generate and validate json schemas. The way this library works, when schemas are generated from a c# class, it marks the schema as not accepting any more properties (NJsonSchema.AllowAdditionalProperties = false).

I have read the following about schema evolution though it seems to be related to AVRO only: https://docs.confluent.io/current/schema-registry/avro.html#schema-evolution-and-compatibility

Given the default compatibility is "BACKWARD" and it says: Changes allowed (Delete fields, Add optional fields), does this mean that the Confluent Schema Registry compatibility type does not work or is not implemeted yet for JSON serialization?

PS: I can see that 2 days ago, the constructor of JsonSerializer was changed to allow setting JsonSchemaGeneratorSettings and potentially override this behavior by setting the AlwaysAllowAdditionalObjectProperties property. However, I would still like to ask whether the compatibility setting is being enforced when using JSON or if we have to manage this "outside" kafka by manipulating the JsonSchemaGeneratorSettings settings.

Using: nuget 1.5.0 and confluentinc/cp-kafka:latest

How to reproduce

Checklist

Please provide the following information:

• [ ] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• [x] Confluent.Kafka nuget version.
• [x] Apache Kafka version.
• [ ] Client configuration.
• [ ] Operating system.
• [ ] Provide logs (with "debug" : "..." as necessary in configuration).
• [ ] Provide broker log excerpts.
• [ ] Critical issue.

[mhowlett]

https://github.com/confluentinc/confluent-kafka-dotnet/pull/1373 will allow you to get around this I believe (not yet in a release, but you could use the appveyor feed). I'm thinking we should probably set AllowAdditionalProperties to true by default.

[devrios]

Thanks @mhowlett , I have used that "yet to be release" version that contains that and I always set it to true for now.

On the basis of the "Compatibility Types" (link above), it may not always be safe to always set it to true. This is because, depending on the compatibility type, it may allow optional fields only (BACKWARD, BACKWARD_TRANSITIVE, FULL and FULL_TRANSITIVE) or any required/optional field (FORWARD, FORWARD_TRANSITIVE and NONE).

To determine if a field is required, you use the "System.ComponentModel.DataAnnotations.Required" attribute. On the other hand if you dont set that attribute, it is marked as optional in the NSchema library.

Bottom line, as far as I can see from the most recent version of "confluent-kafka-dotnet", it is not enforcing/implementing the compatibility types, can you please confirm that?

If you think it is useful to the project, please tell me and I can have a go to enforce the compatibility types and submit a PR.

[mhowlett]

The compatibility check happens in Schema Registry (and I don't have any reason to believe there's a problem there).

[devrios]

I see your point, but I think there is something that needs tuning here. To be clear, I was referring to the client, not kafka schema registry.

The JsonSerializer does never get the schema from the schema registry, it creates its own, locally based on the defaults of the NJsonSchema library. Instead, it should read the compatibility type from the schema registry and set the schema validation according to the respective type so it passes the local validation in order to send the message.

As it stands, messages never get to kafka because the local validation is too restrictive and oblivious to the schema registry.

Please let me know if I am misunderstanding something but this is in fact what raised this issue in the first place.

[jfloodnet]

Yep this is still the case. Basically this library is unusable.