search

amient / affinity

Library and a Framework for building fast, scalable, fault-tolerant Data APIs based on Akka, Avro, ZooKeeper and Kafka
Apache License 2.0
26 stars 4 forks source link

ZookeeperSchemaRegistry #282

Closed gr211 closed 4 years ago

gr211 commented 4 years ago

on https://github.com/amient/affinity/blob/master/avro/src/main/scala/io/amient/affinity/avro/ZookeeperSchemaRegistry.scala#L97

Instead, should the validate method not be conditional somehow to the actual current compatibility settings of the registry itself (ie. check with https://docs.confluent.io/current/schema-registry/develop/api.html#post--compatibility-subjects-(string-%20subject)-versions-(versionId-%20version) ?

Or am I missing the intention on that line L97 ?

I did try to forcefully register a known incompatible schema into a schema registy in NONE compatibility level mode, to no avail:

[error]    org.apache.avro.SchemaValidationException: Unable to read schema: 
[error]    {
[error]      "type" : "record",
[error]      "name" : "EventProduct2",
[error]      "namespace" : "com.model",
[error]      "fields" : [ {
[error]        "name" : "id2",
[error]        "type" : "string"
[error]      } ]
[error]    }
[error]    using schema:
[error]    {
[error]      "type" : "record",
[error]      "name" : "EventProduct",
[error]      "namespace" : "com.model",
[error]      "fields" : [ {
[error]        "name" : "id",
[error]        "type" : "long"
[error]      } ]
[error]    } (ValidateMutualRead.java:70)
[error] org.apache.avro.ValidateMutualRead.canRead(ValidateMutualRead.java:70)
[error] org.apache.avro.ValidateMutualRead.validate(ValidateMutualRead.java:43)
[error] org.apache.avro.ValidateAll.validate(ValidateAll.java:51)
[error] io.amient.affinity.avro.ZookeeperSchemaRegistry.$anonfun$registerSchema$4(ZookeeperSchemaRegistry.scala:97)
[error] io.amient.affinity.avro.ZookeeperSchemaRegistry.$anonfun$registerSchema$1(ZookeeperSchemaRegistry.scala:96)

Relevant case classes are (for test purposes):

trait TestEvent extends AvroRecord

case class EventProduct(id: Long) extends TestEvent
case class EventProduct2(id2: String) extends TestEvent

I understand this is a rather contrived example. However, I think the problem is larger that just this (is actual non compatible schemas should be let in if that is so the wish of the schema registry). Yet before I attempt at providing a potential fix, I wanted to makde sure I understood the current logic right.

gr211 commented 4 years ago

Closing. Realised I was using the wrong schema registry implementation.