search

tchiotludo / akhq

Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more...
https://akhq.io/
Apache License 2.0
3.34k stars 646 forks source link

Handle broken kafka brokers that don't return all needed value #112

Closed apellegr06 closed 3 years ago

apellegr06 commented 5 years ago

Hello,

I have installed version 0.9 and with one instance I have the following error at startup :

java.util.NoSuchElementException: Topic Partition Offsets 'topic1' partition 2 doesn't exist for group group1

java.util.concurrent.ExecutionException: java.util.NoSuchElementException: Topic Partition Offsets 'topic1' partition 2 doesn't exist for group group1 at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.kafkahq.utils.CompletablePaged.complete(CompletablePaged.java:68) at org.kafkahq.controllers.TopicController.list(TopicController.java:96) at org.kafkahq.controllers.$TopicControllerDefinition$$exec1.invokeInternal(Unknown Source) at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:144) at io.micronaut.context.DefaultBeanContext$BeanExecutionHandle.invoke(DefaultBeanContext.java:2711) at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:295) at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:122) at io.micronaut.http.server.netty.RoutingInBoundHandler.lambda$buildResultEmitter$17(RoutingInBoundHandler.java:1360) at io.reactivex.internal.operators.flowable.FlowableCreate.subscribeActual(FlowableCreate.java:71) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14752) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:37) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14752) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty.subscribeActual(FlowableSwitchIfEmpty.java:32) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14752) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14752) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.internal.operators.flowable.FlowableSwitchMap.subscribeActual(FlowableSwitchMap.java:49) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14752) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14752) at io.reactivex.internal.operators.flowable.FlowableDefer.subscribeActual(FlowableDefer.java:42) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14752) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14752) at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FlowableSwitchIfEmpty.java:71) at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52) at io.micronaut.http.context.ServerRequestContext.lambda$instrument$0(ServerRequestContext.java:68) at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.onComplete(InstrumentedSubscriber.java:112) at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drainLoop(FlowableFlatMap.java:426) at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drain(FlowableFlatMap.java:366) at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onComplete(FlowableFlatMap.java:338) at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52) at io.micronaut.http.context.ServerRequestContext.lambda$instrument$0(ServerRequestContext.java:68) at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.onComplete(InstrumentedSubscriber.java:112) at io.reactivex.internal.operators.maybe.MaybeToFlowable$MaybeToFlowableSubscriber.onComplete(MaybeToFlowable.java:80) at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52) at io.micronaut.http.context.ServerRequestContext.lambda$instrument$0(ServerRequestContext.java:68) at io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onComplete(RxInstrumentedMaybeObserver.java:92) at io.reactivex.internal.operators.maybe.MaybeDoOnEvent$DoOnEventMaybeObserver.onComplete(MaybeDoOnEvent.java:115) at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52) at io.micronaut.http.context.ServerRequestContext.lambda$instrument$0(ServerRequestContext.java:68) at io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onComplete(RxInstrumentedMaybeObserver.java:92) at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe$ElementAtSubscriber.onComplete(FlowableElementAtMaybe.java:102) at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52) at io.micronaut.http.context.ServerRequestContext.lambda$instrument$0(ServerRequestContext.java:68) at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.onComplete(InstrumentedSubscriber.java:112) at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drainLoop(FlowableFlatMap.java:426) at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drain(FlowableFlatMap.java:366) at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onComplete(FlowableFlatMap.java:338) at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52) at io.micronaut.http.context.ServerRequestContext.lambda$instrument$0(ServerRequestContext.java:68) at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.onComplete(InstrumentedSubscriber.java:112) at io.reactivex.internal.operators.flowable.FlowableFromIterable$IteratorSubscription.slowPath(FlowableFromIterable.java:255) at io.reactivex.internal.operators.flowable.FlowableFromIterable$BaseRangeSubscription.request(FlowableFromIterable.java:124) at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onSubscribe(FlowableFlatMap.java:117) at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.onSubscribe(InstrumentedSubscriber.java:75) at io.reactivex.internal.operators.flowable.FlowableFromIterable.subscribe(FlowableFromIterable.java:69) at io.reactivex.internal.operators.flowable.FlowableFromIterable.subscribeActual(FlowableFromIterable.java:47) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14752) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.internal.operators.flowable.FlowableFlatMap.subscribeActual(FlowableFlatMap.java:53) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14752) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe.subscribeActual(FlowableElementAtMaybe.java:36) at io.reactivex.Maybe.subscribe(Maybe.java:4262) at io.micronaut.reactive.rxjava2.RxInstrumentedMaybe.subscribeActual(RxInstrumentedMaybe.java:64) at io.reactivex.Maybe.subscribe(Maybe.java:4262) at io.reactivex.internal.operators.maybe.MaybeDoOnEvent.subscribeActual(MaybeDoOnEvent.java:39) at io.reactivex.Maybe.subscribe(Maybe.java:4262) at io.micronaut.reactive.rxjava2.RxInstrumentedMaybe.subscribeActual(RxInstrumentedMaybe.java:64) at io.reactivex.Maybe.subscribe(Maybe.java:4262) at io.reactivex.internal.operators.maybe.MaybeToFlowable.subscribeActual(MaybeToFlowable.java:45) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14752) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.internal.operators.flowable.FlowableFlatMap.subscribeActual(FlowableFlatMap.java:53) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14752) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty.subscribeActual(FlowableSwitchIfEmpty.java:32) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14752) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14755) at io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher.subscribe(WebMetricsPublisher.java:122) at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14752) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.internal.operators.flowable.FlowableSwitchMap.subscribeActual(FlowableSwitchMap.java:49) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14752) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:37) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14752) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14755) at io.micronaut.http.context.ServerRequestTracingPublisher.lambda$subscribe$0(ServerRequestTracingPublisher.java:52) at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52) at io.micronaut.http.context.ServerRequestTracingPublisher.subscribe(ServerRequestTracingPublisher.java:52) at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14752) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14805) at io.reactivex.Flowable.subscribe(Flowable.java:14752) at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82) at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288) at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253) at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:79) at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:143) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.NoSuchElementException: Topic Partition Offsets 'topic1' partition 2 doesn't exist for group group1 at org.kafkahq.models.ConsumerGroup.lambda$new$1(ConsumerGroup.java:48) at java.util.Optional.orElseThrow(Optional.java:290) at org.kafkahq.models.ConsumerGroup.(ConsumerGroup.java:46) at org.kafkahq.repositories.ConsumerGroupRepository.findByName(ConsumerGroupRepository.java:64) at org.kafkahq.repositories.ConsumerGroupRepository.list(ConsumerGroupRepository.java:37) at org.kafkahq.repositories.ConsumerGroupRepository.findByTopic(ConsumerGroupRepository.java:74) at org.kafkahq.repositories.TopicRepository.findByName(TopicRepository.java:106) at org.kafkahq.repositories.TopicRepository.findByName(TopicRepository.java:91) at org.kafkahq.repositories.TopicRepository.lambda$list$0(TopicRepository.java:52) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

tchiotludo commented 5 years ago

Really weird behavior, difficult to debug offline.

Seems that your kafka cluster is unhealthy.

What I understand from the stack trace :

It's an unexpected behavior that shouldn't happen, why KafkaHQ raise the exception.

Do you have a try kafka-* command (especially kafka-topics & kafka-consumer-groups) that is delivered with kafka server to see if this one works and not kafkahq ? To be honest, I will be surprised if command work and not KafkaHQ since both use the same api. But need a try :smile:

apellegr06 commented 5 years ago

Hi @tchiotludo ,

Thank you for your help, and you're right I tried this command yesterday evening and obtain a timeout :

kafka-consumer-groups.sh  --group group1 --describe --bootstrap-server host1:6667

Error: Executing consumer group command failed due to Failed to get offsets by times in 30000ms
org.apache.kafka.common.errors.TimeoutException: Failed to get offsets by times in 30000ms

But what I don't understand it's why just one consumer group crash all the kafkaHQ, I think that this behaviour could be improve.

tchiotludo commented 5 years ago

I'll try to look if it's possible, but to be honest, seems to be a real hard issue. I can't reproduce on my side, so patch must be blindness with a lot of try catch everywhere. Seems really hard for a border less behavior

apellegr06 commented 4 years ago

I have this behaviour on two brokers, and on the second one this behaviour was raised later.

If I can reproduce the problem I will inform you

xakassi commented 3 years ago

Hi, guys! I have such issue now on a customer env and actually do not know what to do.

AKHQ shows error:

java.util.NoSuchElementException: Topic Partition Offsets 'eso4_logs' partition 1 doesn't exist for group connect-eso4_logs-elasticsearch-sink-connector
    at org.akhq.models.ConsumerGroup.lambda$new$1(ConsumerGroup.java:50)
    at java.base/java.util.Optional.orElseThrow(Unknown Source)
    at org.akhq.models.ConsumerGroup.<init>(ConsumerGroup.java:48)
    at org.akhq.repositories.ConsumerGroupRepository.lambda$findByName$3(ConsumerGroupRepository.java:83)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
    at java.base/java.util.HashMap$ValueSpliterator.forEachRemaining(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
    at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
    at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
    at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
    at org.akhq.repositories.ConsumerGroupRepository.findByName(ConsumerGroupRepository.java:85)
    at org.akhq.repositories.ConsumerGroupRepository.lambda$list$0(ConsumerGroupRepository.java:32)
    at org.akhq.utils.PagedList.of(PagedList.java:70)
    at org.akhq.repositories.ConsumerGroupRepository.list(ConsumerGroupRepository.java:32)
    at org.akhq.controllers.GroupController.list(GroupController.java:73)
    at org.akhq.controllers.$GroupControllerDefinition$$exec1.invokeInternal(Unknown Source)
    at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:146)
    at io.micronaut.context.DefaultBeanContext$BeanExecutionHandle.invoke(DefaultBeanContext.java:2995)
    at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:286)
    at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:122)
    at io.micronaut.http.server.netty.RoutingInBoundHandler.lambda$buildResultEmitter$16(RoutingInBoundHandler.java:1474)
    at io.reactivex.internal.operators.flowable.FlowableCreate.subscribeActual(FlowableCreate.java:71)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:37)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty.subscribeActual(FlowableSwitchIfEmpty.java:32)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableSwitchMap.subscribeActual(FlowableSwitchMap.java:49)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onNext(FlowableFlatMap.java:163)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:68)
    at io.reactivex.internal.subscriptions.DeferredScalarSubscription.complete(DeferredScalarSubscription.java:132)
    at io.reactivex.internal.operators.maybe.MaybeToFlowable$MaybeToFlowableSubscriber.onSuccess(MaybeToFlowable.java:70)
    at io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onSuccess(RxInstrumentedMaybeObserver.java:82)
    at io.reactivex.internal.operators.maybe.MaybeDoOnEvent$DoOnEventMaybeObserver.onSuccess(MaybeDoOnEvent.java:86)
    at io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onSuccess(RxInstrumentedMaybeObserver.java:82)
    at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe$ElementAtSubscriber.onNext(FlowableElementAtMaybe.java:80)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:68)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drainLoop(FlowableFlatMap.java:501)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drain(FlowableFlatMap.java:366)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap$InnerSubscriber.onNext(FlowableFlatMap.java:665)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:68)
    at io.reactivex.internal.subscriptions.DeferredScalarSubscription.complete(DeferredScalarSubscription.java:117)
    at io.reactivex.internal.operators.maybe.MaybeToFlowable$MaybeToFlowableSubscriber.onSuccess(MaybeToFlowable.java:70)
    at io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onSuccess(RxInstrumentedMaybeObserver.java:82)
    at io.reactivex.internal.operators.maybe.MaybeCreate$Emitter.onSuccess(MaybeCreate.java:73)
    at java.base/java.util.Optional.ifPresent(Unknown Source)
    at io.micronaut.security.session.SessionAuthenticationFetcher.lambda$fetchAuthentication$0(SessionAuthenticationFetcher.java:53)
    at io.reactivex.internal.operators.maybe.MaybeCreate.subscribeActual(MaybeCreate.java:45)
    at io.reactivex.Maybe.subscribe(Maybe.java:4290)
    at io.micronaut.reactive.rxjava2.RxInstrumentedMaybe.subscribeActual(RxInstrumentedMaybe.java:53)
    at io.reactivex.Maybe.subscribe(Maybe.java:4290)
    at io.reactivex.internal.operators.maybe.MaybeToFlowable.subscribeActual(MaybeToFlowable.java:45)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onNext(FlowableFlatMap.java:163)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:68)
    at io.reactivex.internal.operators.flowable.FlowableFromIterable$IteratorSubscription.slowPath(FlowableFromIterable.java:236)
    at io.reactivex.internal.operators.flowable.FlowableFromIterable$BaseRangeSubscription.request(FlowableFromIterable.java:124)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onSubscribe(FlowableFlatMap.java:117)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:54)
    at io.reactivex.internal.operators.flowable.FlowableFromIterable.subscribe(FlowableFromIterable.java:69)
    at io.reactivex.internal.operators.flowable.FlowableFromIterable.subscribeActual(FlowableFromIterable.java:47)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap.subscribeActual(FlowableFlatMap.java:53)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe.subscribeActual(FlowableElementAtMaybe.java:36)
    at io.reactivex.Maybe.subscribe(Maybe.java:4290)
    at io.micronaut.reactive.rxjava2.RxInstrumentedMaybe.subscribeActual(RxInstrumentedMaybe.java:53)
    at io.reactivex.Maybe.subscribe(Maybe.java:4290)
    at io.reactivex.internal.operators.maybe.MaybeDoOnEvent.subscribeActual(MaybeDoOnEvent.java:39)
    at io.reactivex.Maybe.subscribe(Maybe.java:4290)
    at io.micronaut.reactive.rxjava2.RxInstrumentedMaybe.subscribeActual(RxInstrumentedMaybe.java:53)
    at io.reactivex.Maybe.subscribe(Maybe.java:4290)
    at io.reactivex.internal.operators.maybe.MaybeToFlowable.subscribeActual(MaybeToFlowable.java:45)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap.subscribeActual(FlowableFlatMap.java:53)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty.subscribeActual(FlowableSwitchIfEmpty.java:32)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14868)
    at io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher.subscribe(WebMetricsPublisher.java:153)
    at io.reactivex.internal.operators.flowable.FlowableSwitchMap$SwitchMapSubscriber.onNext(FlowableSwitchMap.java:129)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:68)
    at io.micronaut.core.async.publisher.CompletableFuturePublisher$CompletableFutureSubscription.lambda$request$0(CompletableFuturePublisher.java:87)
    at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
    at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(Unknown Source)
    at java.base/java.util.concurrent.CompletableFuture.whenComplete(Unknown Source)
    at io.micronaut.core.async.publisher.CompletableFuturePublisher$CompletableFutureSubscription.request(CompletableFuturePublisher.java:81)
    at io.reactivex.internal.operators.flowable.FlowableSwitchMap$SwitchMapSubscriber.request(FlowableSwitchMap.java:162)
    at io.reactivex.internal.operators.flowable.FlowableSwitchMap$SwitchMapSubscriber.request(FlowableSwitchMap.java:162)
    at io.reactivex.internal.subscribers.BasicFuseableSubscriber.request(BasicFuseableSubscriber.java:153)
    at io.reactivex.internal.subscriptions.SubscriptionHelper.deferredSetOnce(SubscriptionHelper.java:202)
    at io.reactivex.internal.subscribers.StrictSubscriber.onSubscribe(StrictSubscriber.java:87)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:54)
    at io.reactivex.internal.subscribers.BasicFuseableSubscriber.onSubscribe(BasicFuseableSubscriber.java:67)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:54)
    at io.reactivex.internal.operators.flowable.FlowableSwitchMap$SwitchMapSubscriber.onSubscribe(FlowableSwitchMap.java:93)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:54)
    at io.reactivex.internal.operators.flowable.FlowableSwitchMap$SwitchMapSubscriber.onSubscribe(FlowableSwitchMap.java:93)
    at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:54)
    at io.micronaut.core.async.publisher.CompletableFuturePublisher.subscribe(CompletableFuturePublisher.java:48)
    at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableSwitchMap.subscribeActual(FlowableSwitchMap.java:49)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableSwitchMap.subscribeActual(FlowableSwitchMap.java:49)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:37)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14868)
    at io.micronaut.http.context.ServerRequestTracingPublisher.lambda$subscribe$0(ServerRequestTracingPublisher.java:52)
    at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:68)
    at io.micronaut.http.context.ServerRequestTracingPublisher.subscribe(ServerRequestTracingPublisher.java:52)
    at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
    at io.reactivex.Flowable.subscribe(Flowable.java:14918)
    at io.reactivex.Flowable.subscribe(Flowable.java:14865)
    at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
    at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288)
    at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253)
    at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:79)
    at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:144)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedRunnable.run(InvocationInstrumenterWrappedRunnable.java:48)
    at java.base/java.lang.Thread.run(Unknown Source)

But Kafka itself shows that it's ok, I can describe topics which are shown as broken on AKHQ:

./kafka-topics.sh --describe --topic eso4_logs --bootstrap-server kafka:9092 --command-config ./adminclient.properties
Topic:eso4_logs PartitionCount:1        ReplicationFactor:1     Configs:segment.bytes=1073741824
        Topic: eso4_logs        Partition: 0    Leader: 2       Replicas: 2     Isr: 2

And yes ./kafka-consumer-groups.sh shows timeouts.

Maybe Kafka reboot will help... But it's very sad, because all other services work fine and it seems these broken groups do not bother any of them and Kafka itself.

As for steps to reproduce... I do not know how these groups became broken, but I know another scenario where AKHQ UI throws error and it's very easy to reproduce. You can deploy Kafka with 6 brokers, create some topics. And then redeploy Kafka with 3 or less brokers. Some topics which partitions were placed on removed brokers become broken, and so do AKHQ UI.

xakassi commented 3 years ago

It would be great to handle errors for topics on UI, show NULL values, for example, for topics and groups which cannot be loaded.

tchiotludo commented 3 years ago

I know and this can be handle better now that the ui is in react. We create consumer group to add some extra informations, in the new ui we need to change to async call that will not break the whole app.

xakassi commented 3 years ago

I have investigated our problem, Kafka reboot did not help. I found out that NFS storage is used for Kafka and there are some strange partition folders in data directory, which cannot be removed... I think it's due to NFS incompatibility with Kafka: https://sbg.technology/2018/07/10/kafka-nfs/ I suppose only full storage cleanup will help us.

And I'm really looking forward for the fix for AKHQ with async call. Thanks for the great work! :)

tchiotludo commented 3 years ago

will keep this one #137 as it the same issue