search

tchiotludo / akhq

Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more...
https://akhq.io/
Apache License 2.0
3.35k stars 650 forks source link

Support only Kafka Connect (without Kafka Cluster) #504

Open joshuagrisham opened 3 years ago

joshuagrisham commented 3 years ago

For one use case which I have, it would actually be really nice to just use the Connect feature of Akhq and ignore everything else.
From what I have found, the only 3 decent web-based options for Connect UI are the Confluent Control Center (which is quite nice but must be licensed for a cost), Akhq, and Landoop kafka-connect-ui.

Between Landoop and Akhq I prefer Akhq hands-down for several reasons including that we are already using it and have people who are comfortable with using and configuring it.

This scenario is a separate team managing a separate Connect instance which is using the main Kafka cluster, so they only thing they need to do is view and manage their connectors--it does not make sense for them to connect to the actual Kafka cluster from this instance of Akhq (we already have one running for that which everyone else uses).

It is actually working fine for this scenario except that every time you open the page and/or log in there is this huge red and scary-looking error message that starts like this:

Internal Server Error: null
java.lang.NullPointerException
    at org.akhq.modules.KafkaModule.lambda$getDefaultsProperties$3(KafkaModule.java:66)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
    at java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
...
...
...

But once you close the error then it is fine and the functionality of the Connect section works great!

In the application configuration I have only configured connect and did not give any configuration for bootstrap.servers or anything else.

Is it possible to ignore this error message so it does not display if only Connect is configured?

Or even one step further could be to actually hide all of the other sections in the menu and the default view that comes up goes directly to Connect if that is the only thing which has a configuration.

tchiotludo commented 3 years ago

Hi,

Thanks for large report !!

I was not aware people can use AKHQ this way :) (as a connect standalone gui). So it's don't surprised me that don't work well :)

Maybe you can share the full stacktrace please ? You truncated it too much to understand the errors.

joshuagrisham commented 3 years ago

Yes sure! Here is the full text of the error:

Internal Server Error: null
java.lang.NullPointerException
    at org.akhq.modules.KafkaModule.lambda$getDefaultsProperties$3(KafkaModule.java:66)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
    at java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
    at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source)
    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
    at java.base/java.util.stream.ReferencePipeline.forEach(Unknown Source)
    at org.akhq.modules.KafkaModule.getDefaultsProperties(KafkaModule.java:65)
    at org.akhq.modules.KafkaModule.getAdminProperties(KafkaModule.java:91)
    at org.akhq.modules.KafkaModule.getAdminClient(KafkaModule.java:100)
    at org.akhq.modules.AbstractKafkaWrapper.listTopics(AbstractKafkaWrapper.java:52)
    at org.akhq.modules.$KafkaWrapperRequestScopeDefinition$$exec2.invokeInternal(Unknown Source)
    at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:146)
    at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:60)
    at org.akhq.modules.$KafkaWrapperRequestScopeDefinition$Intercepted.listTopics(Unknown Source)
    at org.akhq.repositories.TopicRepository.all(TopicRepository.java:72)
    at org.akhq.repositories.TopicRepository.list(TopicRepository.java:64)
    at org.akhq.controllers.TopicController.list(TopicController.java:99)
    at org.akhq.controllers.$TopicControllerDefinition$$exec1.invokeInternal(Unknown Source)
    at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:146)
    at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:469)
    at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:312)
    at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:118)
    at io.micronaut.http.server.netty.RoutingInBoundHandler.lambda$buildResultEmitter$9(RoutingInBoundHandler.java:1352)
    at io.reactivex.internal.operators.flowable.FlowableDefer.subscribeActual(FlowableDefer.java:35)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14868)
    at org.akhq.middlewares.HttpServerAccessLogFilter$HttpServerPublisher.subscribe(HttpServerAccessLogFilter.java:130)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onNext(FlowableFlatMap.java:163)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:59)
    at io.reactivex.internal.subscriptions.DeferredScalarSubscription.complete(DeferredScalarSubscription.java:132)
    at io.reactivex.internal.operators.maybe.MaybeToFlowable$MaybeToFlowableSubscriber.onSuccess(MaybeToFlowable.java:70)
    at io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onSuccess(RxInstrumentedMaybeObserver.java:65)
    at io.reactivex.internal.operators.maybe.MaybeDoOnEvent$DoOnEventMaybeObserver.onSuccess(MaybeDoOnEvent.java:86)
    at io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onSuccess(RxInstrumentedMaybeObserver.java:65)
    at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe$ElementAtSubscriber.onNext(FlowableElementAtMaybe.java:80)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:59)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.tryEmit(FlowableFlatMap.java:282)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap$InnerSubscriber.onNext(FlowableFlatMap.java:663)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:59)
    at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FlowableSwitchIfEmpty.java:59)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:59)
    at io.reactivex.internal.operators.flowable.FlowableSwitchMap$SwitchMapSubscriber.drain(FlowableSwitchMap.java:307)
    at io.reactivex.internal.operators.flowable.FlowableSwitchMap$SwitchMapInnerSubscriber.onNext(FlowableSwitchMap.java:391)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:59)
    at io.reactivex.internal.operators.flowable.FlowableMap$MapSubscriber.onNext(FlowableMap.java:68)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:59)
    at io.reactivex.internal.subscriptions.ScalarSubscription.request(ScalarSubscription.java:55)
    at io.reactivex.internal.subscribers.BasicFuseableSubscriber.request(BasicFuseableSubscriber.java:153)
    at io.reactivex.internal.operators.flowable.FlowableSwitchMap$SwitchMapInnerSubscriber.onSubscribe(FlowableSwitchMap.java:379)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:52)
    at io.reactivex.internal.subscribers.BasicFuseableSubscriber.onSubscribe(BasicFuseableSubscriber.java:67)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:52)
    at io.reactivex.internal.operators.flowable.FlowableJust.subscribeActual(FlowableJust.java:34)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:57)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:37)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.reactivex.internal.operators.flowable.FlowableSwitchMap$SwitchMapSubscriber.onNext(FlowableSwitchMap.java:129)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:59)
    at io.reactivex.internal.subscriptions.ScalarSubscription.request(ScalarSubscription.java:55)
    at io.reactivex.internal.operators.flowable.FlowableSwitchMap$SwitchMapSubscriber.request(FlowableSwitchMap.java:162)
    at io.reactivex.internal.subscriptions.SubscriptionArbiter.setSubscription(SubscriptionArbiter.java:99)
    at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty$SwitchIfEmptySubscriber.onSubscribe(FlowableSwitchIfEmpty.java:51)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:52)
    at io.reactivex.internal.operators.flowable.FlowableSwitchMap$SwitchMapSubscriber.onSubscribe(FlowableSwitchMap.java:93)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:52)
    at io.reactivex.internal.operators.flowable.FlowableJust.subscribeActual(FlowableJust.java:34)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:57)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableSwitchMap.subscribeActual(FlowableSwitchMap.java:49)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty.subscribeActual(FlowableSwitchIfEmpty.java:32)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onNext(FlowableFlatMap.java:163)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:59)
    at io.reactivex.internal.operators.flowable.FlowableFromIterable$IteratorSubscription.slowPath(FlowableFromIterable.java:236)
    at io.reactivex.internal.operators.flowable.FlowableFromIterable$BaseRangeSubscription.request(FlowableFromIterable.java:124)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onSubscribe(FlowableFlatMap.java:117)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:52)
    at io.reactivex.internal.operators.flowable.FlowableFromIterable.subscribe(FlowableFromIterable.java:69)
    at io.reactivex.internal.operators.flowable.FlowableFromIterable.subscribeActual(FlowableFromIterable.java:47)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:57)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap.subscribeActual(FlowableFlatMap.java:53)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe.subscribeActual(FlowableElementAtMaybe.java:36)
    at io.reactivex.Maybe.subscribe(Maybe.java:4290)
    at io.micronaut.reactive.rxjava2.RxInstrumentedMaybe.subscribeActual(RxInstrumentedMaybe.java:53)
    at io.reactivex.Maybe.subscribe(Maybe.java:4290)
    at io.reactivex.internal.operators.maybe.MaybeDoOnEvent.subscribeActual(MaybeDoOnEvent.java:39)
    at io.reactivex.Maybe.subscribe(Maybe.java:4290)
    at io.reactivex.internal.operators.maybe.MaybeToFlowable.subscribeActual(MaybeToFlowable.java:45)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap.subscribeActual(FlowableFlatMap.java:53)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty.subscribeActual(FlowableSwitchIfEmpty.java:32)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14868)
    at io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher.subscribe(WebMetricsPublisher.java:153)
    at io.micronaut.http.server.context.ServerRequestContextFilter.lambda$doFilter$0(ServerRequestContextFilter.java:62)
    at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
    at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288)
    at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253)
    at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:79)
    at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:148)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)
tuananh commented 3 years ago

i got the same issue. any idea how to get it fixed?

tuananh commented 3 years ago

in my case, it was because of a missing Tab in setting :(

tchiotludo commented 3 years ago

@tuananh don't understand ? maybe more information please ?

tuananh commented 3 years ago

@tchiotludo ah the properties fields suppose to be indented but wasn't. it causes the same error as above: null pointer exception and stuff like that right when i open the page /ui

raphaelauv commented 3 years ago

@tchiotludo

With this configuration, the connect page is working but the others throw an error and do not accept an empty field for the brokers

  akhq:
    image: tchiotludo/akhq
    environment:
      AKHQ_CONFIGURATION: |
        akhq:
          connections:
            docker-kafka-server:
              properties:
                bootstrap.servers: ""
              schema-registry:
                url: "http://schema-registry:8081"
              connect:
                - name: "connect_docker"
                  url:  "http://connect:8083"

give ->

Screenshot from 2021-06-11 11-52-29

Internal Server Error: Failed to create new KafkaAdminClient</h4><pre>org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
    at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:499)
    at org.apache.kafka.clients.admin.Admin.create(Admin.java:63)
    at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39)
    at org.akhq.modules.KafkaModule.getAdminClient(KafkaModule.java:100)
    at org.akhq.modules.AbstractKafkaWrapper.listTopics(AbstractKafkaWrapper.java:52)
    at org.akhq.modules.$KafkaWrapperRequestScopeDefinition$$exec2.invokeInternal(Unknown Source)
    at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:146)
    at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:73)
    at org.akhq.modules.$KafkaWrapperRequestScopeDefinition$Intercepted.listTopics(Unknown Source)
    at org.akhq.repositories.TopicRepository.all(TopicRepository.java:65)
    at org.akhq.repositories.TopicRepository.list(TopicRepository.java:57)
    at org.akhq.controllers.TopicController.list(TopicController.java:80)
    at org.akhq.controllers.$TopicControllerDefinition$$exec1.invokeInternal(Unknown Source)
    at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:146)
    at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:474)
    at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:312)
    at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:118)
    at io.micronaut.http.server.netty.RoutingInBoundHandler.lambda$buildResultEmitter$10(RoutingInBoundHandler.java:1363)
    at io.reactivex.internal.operators.flowable.FlowableDefer.subscribeActual(FlowableDefer.java:35)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
    at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288)
    at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253)
    at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:79)
    at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:148)
    at io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedRunnable.run(InvocationInstrumenterWrappedRunnable.java:47)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
    at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:89)
    at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:48)
    at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:455)
    ... 29 more