Apicurio Registry support

[jaehyeon-kim]

Is your proposal related to a problem?

Kafka UI supports the Confluent schema registry. Alternative to it is the Apicurio registry and it supports multiple versions of APIs including the Confluent compatible schema registry - Version 6.

We can use the compatible API (KAFKA_CLUSTERS_0_SCHEMAREGISTRY: http://registry:8080/apis/ccompat/v6) but it requires different API client and converters.

It'll be good if Kafka UI supports the main API of Apicurio - Version 2. For example, Apache Camel supports a wide range of connectors and, when I checked a couple of them, I see they include the Apicurio related JARs. If it is supported, it'll be quite easy to create connectors with it and monitor/manage on the UI.

Describe the solution you'd like

Add another configuration that supports the Apicurio version 2 API.

[Haarolean]

Hi, thanks for the suggestion!

Actually, we've been approached a few times before about apicurio (#1177 #1179) but nobody has previously mentioned that it has a separate cool API.

We'll plan this for the future releases (but probably not the next one considering the bandwidth).

[Haarolean]

https://github.com/provectus/kafka-ui/issues/1621#issuecomment-1078421861

[rmarting]

I am testing the latest version of Kafka-UI with the latest version of Apicurio Registry and I found an issue related with the schema versions endpoint. This integration is described in this repository (as an example of use of all these components): https://github.com/rmarting/kafka-clients-quarkus-sample

The url used to connect to the Apicurio Registry is http://service-registry-service:8080/apis/ccompat/v6 that it is the endpoint for the compatibility with Confluent Registry (the right url for Service Registry is /apis/registry/v2).

Here a screenshot of the issue:

The Kafka-UI logs reports the following stacktrace:

2023-05-04 13:59:52,086 ERROR [reactor-http-epoll-6] o.s.b.a.w.r.e.AbstractErrorWebExceptionHandler: [cb544d04-102]  500 Server Error for HTTP GET "/api/clusters/my-cluster/schemas/messages/versions"
org.springframework.web.reactive.function.client.WebClientResponseException$InternalServerError: 500 Internal Server Error from GET http://service-registry-service:8080/apis/ccompat/v6/subjects/messages/versions
    at org.springframework.web.reactive.function.client.WebClientResponseException.create(WebClientResponseException.java:331)
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
    *__checkpoint ⇢ 500 INTERNAL_SERVER_ERROR from GET http://service-registry-service:8080/apis/ccompat/v6/subjects/messages/versions [DefaultWebClient]
    *__checkpoint ⇢ Handler com.provectus.kafka.ui.controller.SchemasController#getAllVersionsBySubject(String, String, ServerWebExchange) [DispatcherHandler]
    *__checkpoint ⇢ com.provectus.kafka.ui.config.ReadOnlyModeFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ com.provectus.kafka.ui.config.CustomWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ AuthorizationWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ ExceptionTranslationWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ LogoutWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ ServerRequestCacheWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ ReactorContextWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ HttpHeaderWriterWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
    *__checkpoint ⇢ org.springframework.web.filter.reactive.ServerHttpObservationFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ HTTP GET "/api/clusters/my-cluster/schemas/messages/versions" [ExceptionHandlingWebHandler]

So, the integration and support with Apicurio Registry is not completed, right?

[Haarolean]

@rmarting well, the issue is still open, so... no, it's not :) But we do support SR-compatible API, though.

[Haarolean]

We're gathering opinions here, this issue requires more clarification and justification.

This part doesn't really explain why kafka-ui would need to support another API and why can't people use SR-compatible API:

It'll be good if Kafka UI supports the main API of Apicurio - Version 2. For example, Apache Camel supports a wide range of connectors and, when I checked a couple of them, I see they include the Apicurio related JARs. If it is supported, it'll be quite easy to create connectors with it and monitor/manage on the UI.

Everyone, please share your use cases.

[alexisph]

One of the reasons why I would like to see support for the Apicurio v2 API are the application properties.

For example, to make kafka-ui work with the SR-compatible API, I had to enable HTTP Basic auth on the Apicurio Service Registry because that is the only auth method supported with the Confluent Schema Registry. Instead, I could have more securely used SSO by setting the Keycloak server URL, client id and client secret as app properties of the Apicurio converter.

I also had to create confluent-compatible schemas (legacy method) and disable writing the artifact ID to Kafka message headers for them to be usable in kafka-ui.

[rmarting]

Apicurio Registry is another Schema Registry with an increasing community, and a lot of features (including many different type of schemas, including new ones). Apicurio Registry provides a Confluent Schema Registry API compatibility, so it is good to integrate with tools like this one (tested and verified), however, it could be great to allow to choose which Schema Registry API you want to use, and not only the Confluent one.

Using the right configuration properties in Kafka-UI, providing an implementation integrated with Apicurio Registry could help use it with a wider kind of schemas registry.

The Apicurio Registry API is more powerfull with a lot of options, that they also could be included in the UI.

Here is the API definition of Apicurio Registry for the 2.4 version (the latest one).

[alexisph]

Hi again. Can something like this be used to integrate Apicurio SR with kafka-ui?

[rmarting]

I tested successfully an integration with Apicurio Registry using the Confluent compatibility API endpoint.

This is the values.yaml file used for my deployment using the Helm Chart:

yamlApplicationConfig:
  kafka:
    clusters:
      - name: my-cluster
        bootstrapServers: my-kafka-kafka-bootstrap.amq-streams-demo.svc:9092
        schemaRegistry: http://service-registry-service:8080/apis/ccompat/v6
  auth:
    type: disabled
  management:
    health:
      ldap:
        enabled: false

I found that it is working smoothly, only one detail about the schemas versioning. Confluent API only use integers (e.g: 1, 2, ...), however Apicurio Registry can use other versioning (e.g: 1.0, 1.1 ...). If your schemas in Apicurio Registry use that versioning mechanism, then Kafka-UI fails with an exception not well managed.

HTH

[bsvicente]

apicurio did an update, now they have it /apis/ccompat/v7