DefaultSchemaResolver resolve schema by content doesn't dereference when auto-register = false, thus failing to find registered schema

[wolfchimneyrock]

Description

Registry Version: 3.0.0.M5 Persistence type: postgresql

I have an avro schema with an enum type used by a kafka producer:

{
  "type": "record",
  "name": "OrderEvent",
  "namespace": "examples.avro",
  "fields": [
    { "name": "ticker", "type": "string" },
    { "name": "company", "type": "string" },
    { "name": "action", "type": { "type": "enum", "name": "Action", "symbols": ["UNKNOWN", "BUY", "SELL"], "default": "UNKNOWN" } },
    { "name": "price", "type": "double" },
    { "name": "quantity", "type": "long" }
  ]
}

when auto-register is enabled, the schema is written to the registry like this:

{"type":"record","name":"OrderEvent","namespace":"examples.avro","fields":[{"name":"ticker","type":"string"},{"name":"company","type":"string"},{"name":"action","type":{"type":"enum","name":"Action","symbols":["UNKNOWN","BUY","SELL"],"default":"UNKNOWN"}},{"name":"price","type":"double"},{"name":"quantity","type":"long"}]

Note the enum type Action is dereferenced.

if I attempt to run the producer with auto-register disabled (the above schema already present in the registry) then it fails. I can see this log:

INFO: Retrieving schema content using string: {"type":"record","name":"OrderEvent","namespace":"examples.avro","fields":[{"name":"ticker","type":"string"},{"name":"company","type":"string"},{"name":"action","type":"Action"},{"name":"price","type":"double"},{"name":"quantity","type":"long"}]}

note the search string is not dereferenced, and thus doesn't match the content in the registry.

and the vertx client logs this response:

{"count":0,"versions":[]}

if auto-register enabled dereferences the schema, then shouldn't auto-register disabled search for the dereferenced version?

[apicurio-bot[bot]]

Thank you for reporting an issue!

Pinging @jsenko to respond or triage.

[wolfchimneyrock]

I think this is the line in question: https://github.com/Apicurio/apicurio-registry/blob/4cebfba6ade04ace1c369b27558c9f312ce40dc3/schema-resolver/src/main/java/io/apicurio/registry/resolver/DefaultSchemaResolver.java#L140

[carlesarnal]

Hello,

This is a bit tricky. Essentially, there are three parts here. The first one, instructing the producer how to register the schema when there are references. Then, if you have a second producer, there is no way for it to know how the schema was registered, dereferenced or not. Finally, the deserializer part, where you have to instruct how you want it to behave, with references or using the dereferenced content. I still have to go through the serdes a bit more before our final release but, if we change this behaviour, and the schema is registered with references, this would fail as well. I'm thinking of introducing either a configuration property allowing to instruct the serializer to send the schema dereferenced. Would that make sense to you? Another option could be to introduce a subsequent call in case the first one fails with the schema dereferenced, as some sort of backup plan.

[carlesarnal]

I have added a PR that solves the problem by introducing a configuration property -> https://github.com/Apicurio/apicurio-registry/pull/5223/files#diff-8b54bbeb908bd07e2a68b7c36601c950347185ce1a543c95f2db4604855d82f2R97

Again, thanks your your reports @wolfchimneyrock, they're extremely helpful now that we're heading the next major release.

[wolfchimneyrock]

I have added a PR that solves the problem by introducing a configuration property -> https://github.com/Apicurio/apicurio-registry/pull/5223/files#diff-8b54bbeb908bd07e2a68b7c36601c950347185ce1a543c95f2db4604855d82f2R97

Again, thanks your your reports @wolfchimneyrock, they're extremely helpful now that we're heading the next major release.

This looks fine to me, although I don't know why you need three different configs for dereferencing - any given client instance would either be a deserializer or an auto-register serializer or a non-auto-register serializer, so wouldn't a single config like you had before (dereference-schema) suffice unambiguously in any of these cases?

also a general question on non-dereferenced registering - does that mean the referenced schemas (in the example above Action) also get auto-registered in the same group? even if the serializer config apicurio.registry.artifact.artifact-id is set to the parent schemas artifactId ?

[carlesarnal]

I have added a PR that solves the problem by introducing a configuration property -> https://github.com/Apicurio/apicurio-registry/pull/5223/files#diff-8b54bbeb908bd07e2a68b7c36601c950347185ce1a543c95f2db4604855d82f2R97 Again, thanks your your reports @wolfchimneyrock, they're extremely helpful now that we're heading the next major release.

This looks fine to me, although I don't know why you need three different configs for dereferencing - any given client instance would either be a deserializer or an auto-register serializer or a non-auto-register serializer, so wouldn't a single config like you had before (dereference-schema) suffice unambiguously in any of these cases?

In https://github.com/Apicurio/apicurio-registry/pull/5200/files I have consolidated the different properties into a single one.

also a general question on non-dereferenced registering - does that mean the referenced schemas (in the example above Action) also get auto-registered in the same group? even if the serializer config apicurio.registry.artifact.artifact-id is set to the parent schemas artifactId ?

They get registered in the same group, yes. The artifact id that is used is usually the name of the schema in avro.