[Feature] Github/Git integration using post hooks

[wtrocki]

APIcurio registry gives developers a lot of features and benefits over schemas in the code, however, one can argue that keeping schemas in the repository and utilizing git versioning/PR building capabilities can replicate some of the capabilities.

To bridge this and convince developers (especially non-Java ones) to use APIcurio across the entire organization, git repository storage could be possibly introduced.

Content bellow is outdatdd See: https://github.com/Apicurio/apicurio-registry/issues/1122#issuecomment-756155876

Challenges

1. Schema storage metadata and human readable parts need to be split (organized)
2. What will happen if the repository is changed outside the Apicurio registry etc.
3. Latency and overall stability of solution
4. Fragile (missing ACID capabilities etc.)

NOTE: I'm happy to work on and contribute this change

Solution

Storage can utilize branches that will have content saved using proper extension and additional metadata json/yaml file:

https://github.com/Apicurio/apicurio-registry/blob/0110b1c7f074f1cca80d9db25021b3ee9eff4d92/utils/sql/src/main/java/io/apicurio/registry/storage/impl/sql/AbstractSqlRegistryStorage.java#L381-L392

Versioning could create separate branches. Repo needs to be dedicated to apicurio storage (there is no way to mix repository with other artifacts.

[EricWittmann]

I think for small scale deployments this could be a reasonable solution. For scalable deployments I'm not sure this is a good approach.

[wtrocki]

@EricWittmann I have actually attempted to implement it now and see how bad an idea it would be. Really sorry for wasting your time. While saving artifacts is simple but additional metadata needs relationships and things that will be really hard to replicate in git repo and thus invalidate benefits of people seeing schemas.

Reads would be very slow overall etc.

[wtrocki]

As food for though I will see if we can improve user experience by

• file import and export
• vscode plugins
• github bots
• Github actions actions

Using storage was a really bad attempt

[alesj]

Well, it doesn’t seem so bad idea to me.

It might be slow, but imho it would be interesting to see an implementation.

Did you completely gave up? If yes, perhaps I would give it a try. :-)

On Wed, 6 Jan 2021 at 22:21, Wojtek Trocki notifications@github.com wrote:

As food for though I will see if we can improve user experience by

• file import and export
• vscode plugins
• github bots
• Github actions actions

Using storage was a really bad attempt

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Apicurio/apicurio-registry/issues/1122#issuecomment-755717602, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACRA6DWNWX7VEWEUDIHZSLSYTH7DANCNFSM4VXLQ3LA .

[EricWittmann]

I should point out that @famartinrh recently added support in the Registry to fire events when interesting things happen, such as creation of new artifacts, updating existing artifacts, etc. The goal is to facilitate integrations with other systems without doing it directly for each target. So if the goal here is to integrate with GitHub in some way, perhaps the event mechanism is the right approach.

[wtrocki]

@EricWittmann +100

Actually when we think about this that would be really exactly what resolves this problem. It was really XYproblem with using storage.

This gives the opportunity to keep repositories, documents etc. in sync and filter out other metadata that needs to be stored but is not something that should be human readable.

[famarting]

This looks interesting I'm open to implement some integration app, camel route or whatever...

[wtrocki]

Would it make sense to introduce another abstraction in the UI for specifying hooks?

There are two cases here:

1. Github repository as source of artifacts (so someone configured to receive hook/update when someone made update of some specific file on master branch etc.) Very similar to how OpenShift builds work: https://docs.openshift.com/container-platform/3.9/dev_guide/builds/triggering_builds.html

This can be done currently in CI/CD using Apicurio CLI (CLI needs to be released and some documentation added to get there)

2. Registry as source that can push data to some form of webhook (so devs can configure builds).

I think Github integration itself is not that useful to the wider community as they might use attlasian, gitlab etc.

[EricWittmann]

Yes I would think this is the longer term goal for the event based integration support: that users could configure some webhooks or something on a per-artifact basis. Would you agree @famartinrh ?