airyhq / airy

💬 Open Source App Framework to build streaming apps with real-time data - 💎 Build real-time data pipelines and make real-time data universally accessible - 🤖 Join historical and real-time data in the stream to create smarter ML and AI applications. - ⚡ Standardize complex data ingestion and stream data to apps with pre-built connectors
https://airy.co/docs/core
Apache License 2.0
366 stars 45 forks source link

Create a component to manage different Schema Registry Providers #4141

Closed AitorAlgorta closed 5 months ago

AitorAlgorta commented 5 months ago

Is your feature request related to a problem? Please describe.

Currently, our application is limited to a single Schema Registry provider, which restricts flexibility and scalability. As our system grows, we foresee the need to integrate with multiple Schema Registry providers to accommodate different requirements and use cases.

Describe the solution you'd like

We propose creating a new component within our application dedicated to managing different Schema Registry providers. This component will abstract away the specifics of each provider, allowing seamless integration with various registries without modifying the core functionality of our application.

Describe alternatives you've considered

One alternative is to continue using our existing approach of directly interfacing with a single Schema Registry provider. However, this approach is not scalable and may lead to increased complexity as we expand our system.

Another alternative is to develop separate modules for each Schema Registry provider. While this approach provides flexibility, it also introduces code duplication and maintenance overhead.

Additional context

Our application is built using Karapace. The new component should support common Schema Registry providers such as Confluent, and others. It should provide a unified interface for interacting with different providers, including functionalities for schema registration, versioning, and compatibility checks.

This feature is crucial for our long-term scalability and interoperability goals. We aim to ensure that our application remains adaptable to evolving requirements and can seamlessly integrate with diverse ecosystem components.