Closed ankon closed 2 years ago
confirming this, multiple tabs are ok though ...
I'd suggest to test with a local kafka. it could be related to the network connecting to kafka, or?
In our case: Pretty sure the problem isn't Kafka or the network to it:
I'm also almost sure that I must break the event loop of micronaut by one way
We're also experiencing the same issue with Live Tail. It absolutely kills the application for everyone else. Even tried increasing Xmx to 6g but it isn't helping.
Any other JVM options that might help ease this situation?
Is it possible to disable the live-tail feature?
Hi there.
I never found a way to reproduce the issue .... I made a small change that can lead to better result but not really sure about that.
Can you try with the dev
version please ?
Hi there.
I never found a way to reproduce the issue .... I made a small change that can lead to better result but not really sure about that.
Can you try with the
dev
version please ?
Just tried with the dev version. The UI is slightly different with selecting a topic, but the result is the same. It freezes up the website for all other users.
A couple of questions/concerns:
Thank you for being responsive. It is appreciated.
To be honest, my way is more to understand & fix the bug than to disable the feature :smile: Maybe some debug log could help :
/metrics
endpoingMaybe I can found some information looking at that :pray:
We have the same issue ... running AWS MSK and AKHQ on Kubernetes (Openshift) ... small difference ... it hangs even with only one tab open
@mwesox Maybe you can provide some log & metrics to help me to understand ?
2020-07-07 20:12:45,594 ERROR pGroup-1-3 o.a.c.ErrorController Error for List topics
--
| java.lang.RuntimeException: Error for List topics
| at org.akhq.utils.Logger.call(Logger.java:26)
| at org.akhq.modules.AbstractKafkaWrapper.listTopics(AbstractKafkaWrapper.java:52)
| at org.akhq.modules.$KafkaWrapperRequestScopeDefinition$$exec2.invokeInternal(Unknown Source)
| at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:146)
| at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:60)
| at org.akhq.modules.$KafkaWrapperRequestScopeDefinition$Intercepted.listTopics(Unknown Source)
| at org.akhq.repositories.TopicRepository.all(TopicRepository.java:71)
| at org.akhq.repositories.TopicRepository.list(TopicRepository.java:63)
| at org.akhq.controllers.TopicController.list(TopicController.java:121)
| at org.akhq.controllers.$TopicControllerDefinition$$exec1.invokeInternal(Unknown Source)
| at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:146)
| at io.micronaut.context.DefaultBeanContext$BeanExecutionHandle.invoke(DefaultBeanContext.java:2995)
| at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:286)
| at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:122)
| at io.micronaut.http.server.netty.RoutingInBoundHandler.lambda$buildResultEmitter$16(RoutingInBoundHandler.java:1474)
| at io.reactivex.internal.operators.flowable.FlowableCreate.subscribeActual(FlowableCreate.java:71)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:37)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty.subscribeActual(FlowableSwitchIfEmpty.java:32)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.internal.operators.flowable.FlowableSwitchMap.subscribeActual(FlowableSwitchMap.java:49)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.reactivex.internal.operators.flowable.FlowableDefer.subscribeActual(FlowableDefer.java:42)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FlowableSwitchIfEmpty.java:71)
| at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onComplete(RxInstrumentedSubscriber.java:97)
| at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drainLoop(FlowableFlatMap.java:426)
| at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drain(FlowableFlatMap.java:366)
| at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onComplete(FlowableFlatMap.java:338)
| at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onComplete(RxInstrumentedSubscriber.java:97)
| at io.reactivex.internal.operators.maybe.MaybeToFlowable$MaybeToFlowableSubscriber.onComplete(MaybeToFlowable.java:80)
| at io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onComplete(RxInstrumentedMaybeObserver.java:96)
| at io.reactivex.internal.operators.maybe.MaybeDoOnEvent$DoOnEventMaybeObserver.onComplete(MaybeDoOnEvent.java:115)
| at io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onComplete(RxInstrumentedMaybeObserver.java:96)
| at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe$ElementAtSubscriber.onComplete(FlowableElementAtMaybe.java:102)
| at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onComplete(RxInstrumentedSubscriber.java:97)
| at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drainLoop(FlowableFlatMap.java:426)
| at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drain(FlowableFlatMap.java:366)
| at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onComplete(FlowableFlatMap.java:338)
| at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onComplete(RxInstrumentedSubscriber.java:97)
| at io.reactivex.internal.operators.flowable.FlowableFromIterable$IteratorSubscription.slowPath(FlowableFromIterable.java:255)
| at io.reactivex.internal.operators.flowable.FlowableFromIterable$BaseRangeSubscription.request(FlowableFromIterable.java:124)
| at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onSubscribe(FlowableFlatMap.java:117)
| at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:54)
| at io.reactivex.internal.operators.flowable.FlowableFromIterable.subscribe(FlowableFromIterable.java:69)
| at io.reactivex.internal.operators.flowable.FlowableFromIterable.subscribeActual(FlowableFromIterable.java:47)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.internal.operators.flowable.FlowableFlatMap.subscribeActual(FlowableFlatMap.java:53)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe.subscribeActual(FlowableElementAtMaybe.java:36)
| at io.reactivex.Maybe.subscribe(Maybe.java:4290)
| at io.micronaut.reactive.rxjava2.RxInstrumentedMaybe.subscribeActual(RxInstrumentedMaybe.java:53)
| at io.reactivex.Maybe.subscribe(Maybe.java:4290)
| at io.reactivex.internal.operators.maybe.MaybeDoOnEvent.subscribeActual(MaybeDoOnEvent.java:39)
| at io.reactivex.Maybe.subscribe(Maybe.java:4290)
| at io.micronaut.reactive.rxjava2.RxInstrumentedMaybe.subscribeActual(RxInstrumentedMaybe.java:53)
| at io.reactivex.Maybe.subscribe(Maybe.java:4290)
| at io.reactivex.internal.operators.maybe.MaybeToFlowable.subscribeActual(MaybeToFlowable.java:45)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.internal.operators.flowable.FlowableFlatMap.subscribeActual(FlowableFlatMap.java:53)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty.subscribeActual(FlowableSwitchIfEmpty.java:32)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14868)
| at io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher.subscribe(WebMetricsPublisher.java:153)
| at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.internal.operators.flowable.FlowableSwitchMap.subscribeActual(FlowableSwitchMap.java:49)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:37)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14868)
| at io.micronaut.http.context.ServerRequestTracingPublisher.lambda$subscribe$0(ServerRequestTracingPublisher.java:52)
| at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:68)
| at io.micronaut.http.context.ServerRequestTracingPublisher.subscribe(ServerRequestTracingPublisher.java:52)
| at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
| at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288)
| at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253)
| at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:79)
| at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:144)
| at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
| at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
| at io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedRunnable.run(InvocationInstrumenterWrappedRunnable.java:48)
| at java.base/java.lang.Thread.run(Unknown Source)
| Caused by: org.apache.kafka.common.config.ConfigException: Missing required configuration "bootstrap.servers" which has no default value.
| at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:477)
| at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:467)
| at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:108)
| at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:142)
| at org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:208)
| at org.apache.kafka.clients.admin.Admin.create(Admin.java:59)
| at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39)
| at org.akhq.modules.KafkaModule.getAdminClient(KafkaModule.java:100)
| at org.akhq.modules.AbstractKafkaWrapper.lambda$listTopics$1(AbstractKafkaWrapper.java:53)
| at org.akhq.utils.Logger.call(Logger.java:19)
| ... 122 common frames omitted
| 2020-07-07 20:12:51,810 ERROR pGroup-1-3 o.a.c.ErrorController Error for List topics
| java.lang.RuntimeException: Error for List topics
| at org.akhq.utils.Logger.call(Logger.java:26)
| at org.akhq.modules.AbstractKafkaWrapper.listTopics(AbstractKafkaWrapper.java:52)
| at org.akhq.modules.$KafkaWrapperRequestScopeDefinition$$exec2.invokeInternal(Unknown Source)
| at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:146)
| at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:60)
| at org.akhq.modules.$KafkaWrapperRequestScopeDefinition$Intercepted.listTopics(Unknown Source)
| at org.akhq.repositories.TopicRepository.all(TopicRepository.java:71)
| at org.akhq.repositories.TopicRepository.list(TopicRepository.java:63)
| at org.akhq.controllers.TopicController.list(TopicController.java:121)
| at org.akhq.controllers.$TopicControllerDefinition$$exec1.invokeInternal(Unknown Source)
| at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:146)
| at io.micronaut.context.DefaultBeanContext$BeanExecutionHandle.invoke(DefaultBeanContext.java:2995)
| at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:286)
| at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:122)
| at io.micronaut.http.server.netty.RoutingInBoundHandler.lambda$buildResultEmitter$16(RoutingInBoundHandler.java:1474)
| at io.reactivex.internal.operators.flowable.FlowableCreate.subscribeActual(FlowableCreate.java:71)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:37)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty.subscribeActual(FlowableSwitchIfEmpty.java:32)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.internal.operators.flowable.FlowableSwitchMap.subscribeActual(FlowableSwitchMap.java:49)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.reactivex.internal.operators.flowable.FlowableDefer.subscribeActual(FlowableDefer.java:42)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FlowableSwitchIfEmpty.java:71)
| at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onComplete(RxInstrumentedSubscriber.java:97)
| at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drainLoop(FlowableFlatMap.java:426)
| at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drain(FlowableFlatMap.java:366)
| at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onComplete(FlowableFlatMap.java:338)
| at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onComplete(RxInstrumentedSubscriber.java:97)
| at io.reactivex.internal.operators.maybe.MaybeToFlowable$MaybeToFlowableSubscriber.onComplete(MaybeToFlowable.java:80)
| at io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onComplete(RxInstrumentedMaybeObserver.java:96)
| at io.reactivex.internal.operators.maybe.MaybeDoOnEvent$DoOnEventMaybeObserver.onComplete(MaybeDoOnEvent.java:115)
| at io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onComplete(RxInstrumentedMaybeObserver.java:96)
| at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe$ElementAtSubscriber.onComplete(FlowableElementAtMaybe.java:102)
| at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onComplete(RxInstrumentedSubscriber.java:97)
| at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drainLoop(FlowableFlatMap.java:426)
| at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drain(FlowableFlatMap.java:366)
| at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onComplete(FlowableFlatMap.java:338)
| at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onComplete(RxInstrumentedSubscriber.java:97)
| at io.reactivex.internal.operators.flowable.FlowableFromIterable$IteratorSubscription.slowPath(FlowableFromIterable.java:255)
| at io.reactivex.internal.operators.flowable.FlowableFromIterable$BaseRangeSubscription.request(FlowableFromIterable.java:124)
| at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onSubscribe(FlowableFlatMap.java:117)
| at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:54)
| at io.reactivex.internal.operators.flowable.FlowableFromIterable.subscribe(FlowableFromIterable.java:69)
| at io.reactivex.internal.operators.flowable.FlowableFromIterable.subscribeActual(FlowableFromIterable.java:47)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.internal.operators.flowable.FlowableFlatMap.subscribeActual(FlowableFlatMap.java:53)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe.subscribeActual(FlowableElementAtMaybe.java:36)
| at io.reactivex.Maybe.subscribe(Maybe.java:4290)
| at io.micronaut.reactive.rxjava2.RxInstrumentedMaybe.subscribeActual(RxInstrumentedMaybe.java:53)
| at io.reactivex.Maybe.subscribe(Maybe.java:4290)
| at io.reactivex.internal.operators.maybe.MaybeDoOnEvent.subscribeActual(MaybeDoOnEvent.java:39)
| at io.reactivex.Maybe.subscribe(Maybe.java:4290)
| at io.micronaut.reactive.rxjava2.RxInstrumentedMaybe.subscribeActual(RxInstrumentedMaybe.java:53)
| at io.reactivex.Maybe.subscribe(Maybe.java:4290)
| at io.reactivex.internal.operators.maybe.MaybeToFlowable.subscribeActual(MaybeToFlowable.java:45)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.internal.operators.flowable.FlowableFlatMap.subscribeActual(FlowableFlatMap.java:53)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty.subscribeActual(FlowableSwitchIfEmpty.java:32)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14868)
| at io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher.subscribe(WebMetricsPublisher.java:153)
| at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.internal.operators.flowable.FlowableSwitchMap.subscribeActual(FlowableSwitchMap.java:49)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:37)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14868)
| at io.micronaut.http.context.ServerRequestTracingPublisher.lambda$subscribe$0(ServerRequestTracingPublisher.java:52)
| at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:68)
| at io.micronaut.http.context.ServerRequestTracingPublisher.subscribe(ServerRequestTracingPublisher.java:52)
| at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29)
| at io.reactivex.Flowable.subscribe(Flowable.java:14918)
| at io.reactivex.Flowable.subscribe(Flowable.java:14865)
| at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
| at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288)
| at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253)
| at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:79)
| at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:144)
| at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
| at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
| at io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedRunnable.run(InvocationInstrumenterWrappedRunnable.java:48)
| at java.base/java.lang.Thread.run(Unknown Source)
| Caused by: org.apache.kafka.common.config.ConfigException: Missing required configuration "bootstrap.servers" which has no default value.
| at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:477)
| at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:467)
| at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:108)
| at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:142)
| at org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:208)
| at org.apache.kafka.clients.admin.Admin.create(Admin.java:59)
| at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39)
| at org.akhq.modules.KafkaModule.getAdminClient(KafkaModule.java:100)
| at org.akhq.modules.AbstractKafkaWrapper.lambda$listTopics$1(AbstractKafkaWrapper.java:53)
| at org.akhq.utils.Logger.call(Logger.java:19)
| ... 122 common frames omitted
| 2020-07-07 20:12:56,512 ERROR pGroup-1-2 o.a.c.ErrorController Connection reset by peer
| java.io.IOException: Connection reset by peer
| at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
| at java.base/sun.nio.ch.SocketDispatcher.read(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
| at java.base/sun.nio.ch.SocketChannelImpl.read(Unknown Source)
| at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
| at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
| at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
| at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
| at java.base/java.lang.Thread.run(Unknown Source)
| 2020-07-07 20:12:57,004 ERROR pGroup-1-3 o.a.c.ErrorController Connection reset by peer
| java.io.IOException: Connection reset by peer
| at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
| at java.base/sun.nio.ch.SocketDispatcher.read(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
| at java.base/sun.nio.ch.SocketChannelImpl.read(Unknown Source)
| at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
| at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
| at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
| at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
| at java.base/java.lang.Thread.run(Unknown Source)
| 2020-07-07 20:12:57,234 ERROR pGroup-1-2 o.a.c.ErrorController Connection reset by peer
| java.io.IOException: Connection reset by peer
| at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
| at java.base/sun.nio.ch.SocketDispatcher.read(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
| at java.base/sun.nio.ch.SocketChannelImpl.read(Unknown Source)
| at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
| at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
| at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
| at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
| at java.base/java.lang.Thread.run(Unknown Source)
| 2020-07-07 20:12:57,988 ERROR pGroup-1-2 o.a.c.ErrorController Connection reset by peer
| java.io.IOException: Connection reset by peer
| at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
| at java.base/sun.nio.ch.SocketDispatcher.read(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
| at java.base/sun.nio.ch.SocketChannelImpl.read(Unknown Source)
| at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
| at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
| at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
| at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
| at java.base/java.lang.Thread.run(Unknown Source)
| 2020-07-07 20:12:58,047 ERROR pGroup-1-3 o.a.c.ErrorController Connection reset by peer
| java.io.IOException: Connection reset by peer
| at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
| at java.base/sun.nio.ch.SocketDispatcher.read(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
| at java.base/sun.nio.ch.SocketChannelImpl.read(Unknown Source)
| at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
| at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
| at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
| at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
| at java.base/java.lang.Thread.run(Unknown Source)
| 2020-07-07 20:13:01,705 ERROR pGroup-1-3 o.a.c.ErrorController Connection reset by peer
| java.io.IOException: Connection reset by peer
| at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
| at java.base/sun.nio.ch.SocketDispatcher.read(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
| at java.base/sun.nio.ch.SocketChannelImpl.read(Unknown Source)
| at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
| at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
| at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
| at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
| at java.base/java.lang.Thread.run(Unknown Source)
| 2020-07-07 20:13:17,353 ERROR pGroup-1-3 o.a.c.ErrorController Connection reset by peer
| java.io.IOException: Connection reset by peer
| at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
| at java.base/sun.nio.ch.SocketDispatcher.read(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
| at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
| at java.base/sun.nio.ch.SocketChannelImpl.read(Unknown Source)
| at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
| at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
| at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
| at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
| at java.base/java.lang.Thread.run(Unknown Source)
Metrics
{"names":["executor","executor.active","executor.completed","executor.pool.size","executor.queue.remaining","executor.queued","http.server.requests","jvm.buffer.count","jvm.buffer.memory.used","jvm.buffer.total.capacity","jvm.classes.loaded","jvm.classes.unloaded","jvm.gc.live.data.size","jvm.gc.max.data.size","jvm.gc.memory.allocated","jvm.gc.memory.promoted","jvm.gc.pause","jvm.memory.committed","jvm.memory.max","jvm.memory.used","jvm.threads.daemon","jvm.threads.live","jvm.threads.peak","jvm.threads.states","logback.events","process.cpu.usage","process.files.max","process.files.open","process.start.time","process.uptime","system.cpu.count","system.cpu.usage","system.load.average.1m"]}
So if I understand correctly this is supposedly fixed with 0.15.0?
Not really, it was a try that do better than worst but don't seems to work. I was thinking about it, I've the feeling that is must be for all people that use a session (auth). Maybe the session is locked on micronaut.
@mwesox & @ankon is this your case (authentification enabled) ?
@tchiotludo : You mean authentication to KafkaHQ? It is disabled in our case.
arf thanks @mwesox , wrong feeling ...
and I just see that I forgot a response :beach_umbrella:
I was wrong on the metrics url :
maybe this can help ...
@tchiotludo : Maybe try better to reproduce it yourself ... seems like all users running KafkaHQ on kubernetes have the issue .. so maybe try to setup minishift or so and run it there ? Next step maybe run it on AWS EKS ?
We have the issue, not running on k8s just standalone jar, with ldap Auth.
Op di 4 aug. 2020 18:19 schreef Matthias notifications@github.com:
@tchiotludo https://github.com/tchiotludo : Maybe try better to reproduce it yourself ... seems like all users running KafkaHQ on kubernetes have the issue .. so maybe try to setup minishift or so and run it there ? Next step maybe run it on AWS EKS ?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tchiotludo/akhq/issues/187#issuecomment-668691918, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABPKAGAMWFVED3H4Y5F4K3R7AYHNANCNFSM4KHFVOSQ .
To be honest, I'll try to reproduce myself and I'm really not able to reproduce it ! I'll try with docker / k8s and standalone, it don't seems to be the akhq env that seems to cause the issue. I don't have any kafka cluster on AWS or any AWS account, so can't test it.
The main feeling I have is based on authentication (mostly session locking) but don't seems to be the case.
So clearly I have no clue and no reproduction, If no one send me some logs / metrics / configuration, I will not be able to find the issue.
Same issue for me
I am running akhq dev docker version in ECS behind a reverse proxy using nginx
I will try to getting some logs
@ankon @dinegri @mwesox @jorgheymans @svaranasi-traderev .
I'm not really sure since I can't reproduce the issue even trying to add some Thread.sleep everywhere !
I've made some change about that here : https://github.com/tchiotludo/akhq/commit/60a33bb5965a340c345b323c6e9d690bf270de26
Can you try with dev
image please ?
Can we maybe at least have a config option to disable the live tailing feature? Would you be interested in a PR for that?
@dajudge why disabled ? it's a really issue for you now ?
as using the live tailing feature seems to block further HTTP communication this leads to readiness-/liveness-probes failing for the container in k8s - which in turn makes our gitops tooling unhappy and triggers alerting. tbh we'd be better off to just disable the feature entirely so no user gets tempted to use it and use some alternative like kafkacat or so...
@dajudge : https://github.com/micronaut-projects/micronaut-core/discussions/5080
Hope a response to clearly fix that :smile:
@tchiotludo Thanks for looking into it! Let me know if we can help!
I had the same problem:
My architecture is the following: AWS +Kops +EC2 +kubernetes
When I checked kuvernetes events, I noticed how I had "redinessprobe" problems, looking for the possible causes I found the following kubernetes "bug":
https://github.com/kubernetes/kubernetes/issues/89898
So I provided one of the workarounds they proposed, and changed the httpGet with to a simple exec running curl. This change implied updating the docker image, installing curl, and the helm chart, changing the redinessprobe.
Dockerfile:
FROM tchiotludo/akhq:0.17.0
USER root
RUN apt-get update && apt-get install -y curl \
&& rm -rf /var/lib/apt/lists/*
USER akhq
deployment.yaml:
readinessProbe:
exec:
command:
- "curl"
- "-s"
- "http://localhost:8081{{ .Values.readinessProbe.prefix | default "" }}/health"
With these changes I was able to open several "trails" in several tabs.
On the other hand, and taking advantage of the fact that I had to clone and modify the helm chart, I took the opportunity to customize some parameters of the jvm, since the performance of the application was not as expected, or it was not like that of the on-premises version (without kubernetes)
JVM:
-Xms2048m
-Xmx2048m
kubernetes limits & requests
resources:
limits:
memory: 2.5Gi
requests:
cpu: 1
memory: 2.5Gi
With these changes I was able to enjoy a fully functional installation on kubernetes
@tchiotludo If you think that all this can be of help to you, I can make you a pull request with all the changes that worked for me.
Just for information: same issue on MSK + AWS ALB + ECS on Fargate in our case. We also use Cognito in front of our ALB for Authentication.
here is the final result : https://github.com/micronaut-projects/micronaut-core/issues/5123. It's a bug in micronaut, will wait for the fix and if not coming, will use the workaround
Looks like the fix has been merged in the Micronaut project, not sure tho if this is available in 2.5.5.
@ebrard it's available on the micronaut 2.5.5 and available on dev
, you can test please ? :+1:
I've just tried against a production cluster and it seems indeed to work better. I've tried only with one active tail view, but this was already slowing down every other calls before.
I can reproduce this reliably, but haven't yet had time to look into this further.
In my case the 2nd window will hang without ever rendering anything of KafkaHQ. As soon as a I stop the live tailing in the first window the second window will load.
Possibly relevant stuff: