Apicurio Registry v3 ideas

[jsenko]

NOTE: this epic is deprecated

Please use this instead: https://github.com/orgs/Apicurio/projects/6/views/1 (work in progress)

A list of breaking changes to consider when we start planning for v3:

Cleanup/Refactoring

• [x] Remove the IBM Compatibility API
• [ ] Remove the Confluent Compatibility API v6.
• [x] Remove Apicurio Registry REST API v1
• [ ] Standardize naming and clean up config properties
• [x] #57
• [x] Remove dynamic logging API
• [ ] https://github.com/Apicurio/apicurio-registry/issues/3618

Features/Improvements

• [ ] KafkaSQL snapshotting
• [ ] Caching (https://github.com/Apicurio/apicurio-tasks/issues/1)
• [ ] Customizable authorization rules with application enforcement.
• [ ] Change the Artifact and Versions concepts - Artifact should be a container of versions and have its own metadata
• [ ] Support authorization at the group level
• [ ] Gitops
• [ ] Semantic versioning
• [ ] Replace client with Kiota generated one
• [ ] Create javascript Kiota client (consider using in UI)
• [ ] Apache Flink compatibility (serdes for Flink)
• [ ] Apache Pulsar compatibility (serdes for Pulsar)
• [ ] NATS compatibility (serdes for NATS)
• [x] Data models v2
• [ ] Improve error user experience (https://github.com/Apicurio/apicurio-registry/issues/1929 https://github.com/Apicurio/apicurio-registry/issues/1929 https://github.com/Apicurio/apicurio-registry/issues/1763)
• [ ] Rewrite go CLI in Java as a native quarkus executable (and possibly keep it in the registry repo since we are doing this with the operator), use Kiota

Operator

• [ ] Rewrite in Java
• [ ] https://github.com/Apicurio/apicurio-registry/issues/3472
• [ ] Align version of Apicurio Registry, Apicurio Registry Operator and CSV to the single upstream version. This avoids confusion we've seen from the customers when installing Service Registry Operator. The first two can be aligned by our decision to keep the new Operator code in the Registry repository. The CSV must be able to handle situations where the Operator and Operand versions do not change, such as respins, but we can define a versioning approach for these cases, such as an additional number, e.g. 3.0.0-r1, 3.0.0-r2, ...

Notes/Ideas

• Define behavior around deleting artifacts/versions. Do we want to guarantee that content cannot be deleted, only updated? Consider artifact states as well.
• We also have an option to change in a non-backwards compatible way:
  • KafkaSQL messages
  • SQL table structure
  • Schema resolver / serde interfaces, there are some deprecated classes there
• Use common-app-components, maybe even extract some more existing features to there.
• Cleanup of RegistryStorage interface and related concepts. For example:
  • We have a separate database version entity, but RegistryStorage interface does not have a separate public method for creating versions. They can be only created using the createArtifact* method. This should be separated, and we should have a higher-level layer that does the current functionality of createArtifact*. There is a private method createArtifactVersion that IMHO should not be private. Consider same with content.
  • Split colums such as labels and properties into separate tables, instead of using string serialization (see SqlUtil.serializeProperties method).
  • Separate or consolidate artifact and version metadata, e.g. updateArtifactVersionMetaData takes EditableArtifactMetaDataDto. Consider having a separate metadata table for metadata.
• Consider having canonical conceptual model classes, available in a separate maven artifact. Review the current DTOs, together with the RegistryStorage interface, but this model can be wider and include classes currently in the common module.
• Consider using the SPI module decomposition architecture (we are using it somewhat in the common app components). Define a separate SPI module for a given feature that defines interfaces that need to be implemented by a provider of such feature. Then the app/core module can use CDI to find if/which implementors are on the classpath, and possibly provide default implementations. This alleviates strong module dependencies. For example, define an SPI for artifact type providers, and add them to the classpath as needed instead of the app module directly depending on apicurio-registry-schema-util-* via apicurio-registry-schema-util-provider. This can also be done for storage implementations.
• Review and cleanup of exceptions, related to "Improve error user experience" below.
• When currently updating (editable) artifact metadata, we also behind the scenes update metadata of the latest artifact version. This happens because artifact metadata includes labels and properties, but they have a globalID as a foreign key, which requires attaching them to the latest version of the artifact. This does not seem to be correct, since artifact should conceptually have it's own set of metadata. (Maybe each version inheriting them when returned to user.)
• Institute ordering on artifact versions. Currently, we only know which version is the latest. We have a workaround to get an ordering by looking at globalIDs (assuming they are monotonic), but this is error-prone. We should have a predecessor field in every version, to be able to reconstruct the ordering. This will also facilitate git-like versioning (i.e. allow for git-like version trees).

[EricWittmann]

Data models v2

[jsenko]

We also have an option to change in a non-backwards compatible way:

• KafkaSQL messages
• SQL table structure
• Schema resolver / serde interfaces, there are some deprecated classes there

[andreaTP]

• Kiota based SDK libraries (Java, TS, Python, Go)
• Drop the manually maintained client library

[EricWittmann]

• Release a 2.x with deprecation warnings
• Write up a blog post with the list of deprecated features and instructions about how to complain

[EricWittmann]

• Moveable named tags/pointers

[EricWittmann]

• KafkaSQL snapshotting

[andreaTP]

• Better modularization of registry to enable proper layering https://github.com/Apicurio/apicurio-registry/issues/3618