provectus / kafka-ui

Open-Source Web UI for Apache Kafka Management
Apache License 2.0
9.08k stars 1.11k forks source link

KafkaConnect configuration does not support clustered endpoint #961

Open akamensky opened 2 years ago

akamensky commented 2 years ago

Describe the bug KafkaConnect is meant to run as a cluster with multiple hosts balancing the load using Kafka as communication mechanism (same as SchemaRegistry). Each node provides API endpoint that affects entire cluster. If one node goes down the cluster still functions.

This UI assumes only single endpoint in

Set up Docker image 0.2.1

Steps to Reproduce Steps to reproduce the behavior:

  1. Have multiple KC nodes (in our case 5+)
  2. Try to define KC configuration in envvars in a way that allows taking down KC nodes for maintenance without disrupting UI functionality (in this setup tried http://redacted1:8016,http://redacted2:8016,http://redacted3:8016)
  3. Above is impossible because configuration takes only single host

Expected behavior Can define comma-separated list of hosts (same as some other tools do)

Logs

07:56:06.544 [reactor-http-epoll-2] ERROR org.springframework.boot.autoconfigure.web.reactive.error.AbstractErrorWebExceptionHandler - [37d4100d]  500 Server Error for HTTP GET "/api/clusters/cluster_name/connectors?search="
org.springframework.web.reactive.function.client.WebClientResponseException$NotFound: 404 Not Found from GET http://redacted1:8016/,http:/redacted2:8016,http:/redacted3:8016/connectors
    at org.springframework.web.reactive.function.client.WebClientResponseException.create(WebClientResponseException.java:185) ~[spring-webflux-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
    |_ checkpoint ⇢ 404 from GET http://redacted1:8016/,http:/redacted2:8016,http:/redacted3:8016/connectors [DefaultWebClient]
    |_ checkpoint ⇢ Handler com.provectus.kafka.ui.controller.KafkaConnectController#getAllConnectors(String, String, ServerWebExchange) [DispatcherHandler]
    |_ checkpoint ⇢ com.provectus.kafka.ui.config.CustomWebFilter [DefaultWebFilterChain]
    |_ checkpoint ⇢ com.provectus.kafka.ui.config.ReadOnlyModeFilter [DefaultWebFilterChain]
    |_ checkpoint ⇢ org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain]
    |_ checkpoint ⇢ org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain]
    |_ checkpoint ⇢ org.springframework.security.web.server.authentication.logout.LogoutWebFilter [DefaultWebFilterChain]
    |_ checkpoint ⇢ org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain]
    |_ checkpoint ⇢ org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
    |_ checkpoint ⇢ org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain]
    |_ checkpoint ⇢ org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain]
    |_ checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
    |_ checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
    |_ checkpoint ⇢ HTTP GET "/api/clusters/cluster_name/connectors?search=" [ExceptionHandlingWebHandler]
Stack trace:
        at org.springframework.web.reactive.function.client.WebClientResponseException.create(WebClientResponseException.java:185) ~[spring-webflux-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
        at org.springframework.web.reactive.function.client.DefaultClientResponse.lambda$createException$1(DefaultClientResponse.java:209) ~[spring-webflux-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
        at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:100) ~[reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
        at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:92) ~[reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
        at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121) ~[reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
        at reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:103) ~[reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
        at reactor.core.publisher.FluxContextStart$ContextStartSubscriber.onNext(FluxContextStart.java:103) ~[reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]

Additional context It is same exact issue as #886

Haarolean commented 2 years ago

Thanks for reporting the issue, we'll aim to fix this in 0.4.

Haarolean commented 2 years ago

The implementation could be similar to #886, up for grabs