search

Apicurio / apicurio-registry

An API/Schema registry - stores APIs and Schemas.
https://www.apicur.io/registry/
Apache License 2.0
609 stars 269 forks source link

How to push avro schemas/protocols such that it registry uses references and doesn't embed everything into definition #5573

Open lsegv opened 14 hours ago

lsegv commented 14 hours ago

I want to build a custom registry image based on apicurio, this image would preload all my schemas enforcing them (since i will disable pushing artifacts by services). There is one big problem though.

If i simply loop over all my AVSC files (that i generate from AVDL) and upload them one by one then registry does no processing on top of them and does not recognise when 2 messages were imported reused, because the content of AVSC is literally inlined to be self contained.

I started looking for api that would allow me to upload protocol files AVDL or AVPR, but i found no such thing.

When i use a kafka producer/consumer example and let it push the definitions on the go i see that this pushes messages properly (e.g. uses references), but actual code that does this does a lot of work, it figures out all those references and uploads them correctly to registry.

Problem is i cant just let arbitrary java code run during build stage (or at least i dont want to), ideally registry should allow for importing a protocol and then properly store all those definitions and references.

What can i do here other than grabbing the code from kafka serializer? maybe there is an api i do not know about?

apicurio-bot[bot] commented 14 hours ago

Thank you for reporting an issue!

Pinging @jsenko to respond or triage.

EricWittmann commented 5 hours ago

What are your expectations for how your schemas are laid out locally? Do you have control over that such that you could maintain some extra metadata?

Another place to look for this type of thing is in our maven plugin. Especially this part:

https://github.com/Apicurio/apicurio-registry/blob/main/utils/maven-plugin/src/main/java/io/apicurio/registry/maven/RegisterRegistryMojo.java#L130-L138

What do you mean precisely when you say this?

this image would preload all my schemas enforcing them

I'd like to better understand your use-case/goals.

There is not currently a way to send a bunch of related stuff to registry all at once and have it automatically figure out the details (with references and all that). It is something we've discussed, but not yet implemented. Could be an opportunity to collaborate on something like that if you are interested.