General purpose artifact synchonization library

[jsenko]

Extend the use case of https://github.com/Apicurio/apicurio-registry-content-sync-operator to create an artifact synchronization engine that would be extendable for different use-cases (sources), such as:

• k8s resources (the current use-case of the Operator)
• file system
• another Registry

[jsenko]

Chat thread link: https://chat.google.com/room/AAAAyw20bSc/B7_9-Qp8mfE

[EricWittmann]

We discussed this during a team call today and reached the following conclusions (in no particular order):

One-way sync only

It is difficult and dangerous to implement a two-way sync between a source and a database/registry. And it's incredibly difficult to maintain. We're not sure what the use-case is for a two-way sync between a source and Apicurio Registry. So let's focus on a one-way sync, with the assumption that the "from" is the source of truth and the registry should be updated to reflect that source of truth.

Sync Use-cases

Kubernetes custom resources

Allow k8s custom resources to be the source of truth for all artifacts in the registry. A sync operator will ensure that whatever is in etcd will be reflected in the registry. It's best to

Mirror-maker equivalent

It would be useful if the "source" for the sync is another registry. This would facilitate a pattern where a single deployment of registry acts as a read/write source. Then a synchronizer can sync the content in that deployment with one or more "read-only" registry deployments. This allows interesting HA and scalability patterns.

Gitops

It's possible to enable gitops by having a synchronizer between a collection of files in a git repository and an Apicurio Registry. Whenever the git repo is changed, the synchronizer will ensure that the changes are reflected in registry.

[jsenko]

Just a small note to the last point, syncing files from a git repository is a special case for syncing files in general (e.g. from a disk). We should also consider what the structure of such file repository would look like, e.g. how to specify artifact metadata, group, etc. in addition to the content itself. There are many possibilities, but we should consider sharing some of the k8s CRs structure.

[andreaTP]

A quick note to be aligned, etcd objects limit (e.g. a k8s Secret) is 1Mb, we should keep this into consideration when designing the feature.

[EricWittmann]

We should also consider what the structure of such file repository would look like, e.g. how to specify artifact metadata, group, etc. in addition to the content itself. There are many possibilities, but we should consider sharing some of the k8s CRs structure.

Agreed. We should also consider artifact references. It would be super awesome to support references between the artifacts automatically and properly set up the reference mappings when syncing the content.

[jsenko]

Related discussion from the community https://github.com/Apicurio/apicurio-registry/discussions/2701