search

confluentinc / schema-registry

Confluent Schema Registry for Kafka
https://docs.confluent.io/current/schema-registry/docs/index.html
Other
2.22k stars 1.11k forks source link

💡 maven schema-registry:register all #1914

Open mac2000 opened 3 years ago

mac2000 commented 3 years ago

In case if we do not want to have all 100+ schemas listed in the subjects section of the plugin configuration

Will be so nice if the plugin will just use everything it can find under outputDirectory if the subjects section missing

image

OneCricketeer commented 3 years ago

Similar to #690

OneCricketeer commented 3 years ago

One problem with this is that AVDL record schemas cannot have hyphens, so using TopicNameStrategy registration isn't possible

mac2000 commented 3 years ago

@OneCricketeer not sure if I got you right, but after downloading I got a bunch of TopicA-key.avcs, TopicA-value.avcs files, so not sure about AVDL files but, if starting from scratch everything should work fine.

OneCricketeer commented 3 years ago

IDL files generate AVSC and can more easily be used for cross-referenced schemas with import support and composition

https://avro.apache.org/docs/current/idl.html

I'm talking about compilation / resource processing via the Avro Maven plugin, not related to the Registry

ena1106 commented 2 years ago

Using TopicRecordNameStrategy the list of subject-schema mapping is not just long but also unreadable.

<schemaTypes>
     <my-topic-com.mycompany.avro.schemas.test.Event1>AVRO</my-topic-com.mycompany.avro.schemas.test.Event1>
     <my-topic-com.mycompany.avro.schemas.test.Event2>AVRO</my-topic-com.mycompany.avro.schemas.test.Event2>
</schemaTypes>

It would also be nice to have a schemaType like AVRO-IDL to check schema compatibility from idl files. There is no official maven plugin to convert from avdl files to avsc. The avro-maven-plugin with goal idl-protocol lets you generate classes directly from avdl files. With this setup you don't have avsc files in the project.

Given these two limitations (long mapping list and lack of avdl support) I think this plugin is unusable in a production environment.

Please tell me if I am missing something.

OneCricketeer commented 2 years ago

The idl-protocol goal does create AVSC files.

In the past, we used it with the exec plugin to define a class and maven goal that iterates over AVDL or AVSC and registers them to the registry

ena1106 commented 2 years ago

Can you please tell me where are the AVSC files generated by the idl-protocol goal? I can only see classes under target/generated-sources/avro/classes

Maybe the exec plugin was getting the schemas from the classes? The json string is present in a static variable of generated classes.

OneCricketeer commented 2 years ago

Well within each class, there's a static SCHEMA$ field that has that information.

But this is what I was referring to http://www.virtualroadside.com/blog/index.php/2014/06/08/automatically-generating-avro-schemata-avsc-files-using-maven/

ena1106 commented 2 years ago

I already read that blog post. They generate avsc files with maven-exec and Avro tools library, not using the idl-protocol goal of the avro-maven-plugin.

OneCricketeer commented 2 years ago

Does that matter? I just couldn't remember which plugins we used. Based on what I have posted, I think we used the exec plugin followed by the avro plugin's schema goal to generate classes, then you can also use Schema Registry maven plugin to register those AVSC as well