search

tchiotludo / akhq

Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more...
https://akhq.io/
Apache License 2.0
3.4k stars 657 forks source link

Akhq stop working. #250

Closed MCPDanilovich closed 4 years ago

MCPDanilovich commented 4 years ago

We have been using akhq for 3 mounths. Our problem that akhq stops work at different points in time. Akhq works in k8s cluster and published via ingress. Sometimes i get 504 error.

My config: `application.yaml: |- akhq: server: base-path: "" access-log: enabled: true name: org.akhq.log.access format: "[Date: {}] [Duration: {} ms] [Url: {} {} {}] [Status: {}] [Ip: {}] [Length: {}] [Port: {}]"

  clients-defaults:
    consumer:
      properties:
        isolation.level: read_committed
  connections:
    kafka-gce-rc:
      properties:
        bootstrap.servers: "servers list"
  security:
    default-group:
      - node-read
    groups:
      node-read:
        roles:
           - node/read
        attributes:
        # Regexp to filter topic available for group
          topics-filter-regexp: ".*"
      kafkahq-reader: # Group name
        roles:  # roles for the group
           - topic/read
           - node/read
           - topic/data/read
           - group/read
           - registry/read
           - connect/read
        attributes:
        # Regexp to filter topic available for group
          topics-filter-regexp: ".*"
      kafkahq-admin:
        roles:
          - topic/read
          - topic/insert
          - topic/delete
          - topic/config/update
          - node/read
          - node/config/update
          - topic/data/read
          - topic/data/insert
          - topic/data/delete
          - group/read
          - group/delete
          - group/offsets/update
          - acls/read
          - registry/read
          - registry/insert
          - registry/update
          - registry/delete
          - registry/version/delete
        attributes:
          topics-filter-regexp: ".*"
    ldap:
      group:
        kafkahq-user:
          groups:
            - kafkahq-reader
        ipausers:
          groups:
            - kafkahq-admin
        kafkahq-admin:
          groups:
            - kafkahq-admin
  pagination:
    page-size: 15 # number of elements per page (default : 25)
    threads: 20 # Number of parallel threads to resolve page
  topic:
    retention: 172800000 # default retention period when creating topic
    partition: 3 #  default number of partition when creating topic
    replication: 3 # default number of replicas when creating topic
    default-view: HIDE_INTERNAL # default list view (ALL, HIDE_INTERNAL, HIDE_INTERNAL_STREAM, HIDE_STREAM)
    skip-consumer-groups: true
    internal-regexps: # list of regexp to be considered as internal (internal topic can't be deleted or updated)
      - "^_.*$"
      - "^.*_schemas$"
      - "^.*connect-config$"
      - "^.*connect-offsets$1"
      - "^.*connect-status$"
    stream-regexps: # list of regexp to be considered as internal stream topic
      - "^.*-changelog$"
      - "^.*-repartition$"
      - "^.*-rekey$"
  topic-data:
    sort: OLDEST # default sort order (OLDEST, NEWEST) (default: OLDEST)
    size: 50 # max record per page (default: 50)
    poll-timeout: 29900 # The time, in milliseconds, spent waiting in poll if data is not available in the buffer.

Security config: micronaut: security: ldap: default: enabled: true context: server: 'ldap://ip-address:389' managerDn: 'uid=service-kafka-hq,cn=users,cn=accounts,dc=,dc=' managerPassword: 'password' search: base: "cn=users,cn=accounts,dc=,dc=" groups: subtree: true enabled: true base: "cn=groups,cn=accounts,dc=,dc=" filter: "member={0}"`

Error: [Date: 2020-03-20T11:09:24.231255Z] [Duration: 1 ms] [Url: GET /health HTTP/1.1] [Status: 200] [Ip: 10.28.3.1] [Length: 215] [Port: 8080] 2020-03-20 11:10:06,095 ERROR pGroup-1-2 o.a.c.ErrorController Error for List topics java.lang.RuntimeException: Error for List topics at org.akhq.utils.Logger.call(Logger.java:26) at org.akhq.modules.AbstractKafkaWrapper.listTopics(AbstractKafkaWrapper.java:51) at org.akhq.modules.$KafkaWrapperRequestScopeDefinition$$exec2.invokeInternal(Unknown Source) at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:146) at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:60) at org.akhq.modules.$KafkaWrapperRequestScopeDefinition$Intercepted.listTopics(Unknown Source) at org.akhq.repositories.TopicRepository.all(TopicRepository.java:70) at org.akhq.repositories.TopicRepository.list(TopicRepository.java:62) at org.akhq.controllers.TopicController.list(TopicController.java:118) at org.akhq.controllers.$TopicControllerDefinition$$exec7.invokeInternal(Unknown Source) at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:146) at io.micronaut.context.DefaultBeanContext$BeanExecutionHandle.invoke(DefaultBeanContext.java:2980) at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:295) at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:122) at io.micronaut.http.server.netty.RoutingInBoundHandler.lambda$buildResultEmitter$15(RoutingInBoundHandler.java:1469) at io.reactivex.internal.operators.flowable.FlowableCreate.subscribeActual(FlowableCreate.java:71) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:37) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty.subscribeActual(FlowableSwitchIfEmpty.java:32) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.internal.operators.flowable.FlowableSwitchMap.subscribeActual(FlowableSwitchMap.java:49) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onNext(FlowableFlatMap.java:163) at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:70) at io.reactivex.internal.subscriptions.DeferredScalarSubscription.complete(DeferredScalarSubscription.java:132) at io.reactivex.internal.operators.maybe.MaybeToFlowable$MaybeToFlowableSubscriber.onSuccess(MaybeToFlowable.java:70) at io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onSuccess(RxInstrumentedMaybeObserver.java:85) at io.reactivex.internal.operators.maybe.MaybeDoOnEvent$DoOnEventMaybeObserver.onSuccess(MaybeDoOnEvent.java:86) at io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onSuccess(RxInstrumentedMaybeObserver.java:85) at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe$ElementAtSubscriber.onNext(FlowableElementAtMaybe.java:80) at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:70) at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drainLoop(FlowableFlatMap.java:501) at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drain(FlowableFlatMap.java:366) at io.reactivex.internal.operators.flowable.FlowableFlatMap$InnerSubscriber.onNext(FlowableFlatMap.java:665) at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:70) at io.reactivex.internal.subscriptions.DeferredScalarSubscription.complete(DeferredScalarSubscription.java:117) at io.reactivex.internal.operators.maybe.MaybeToFlowable$MaybeToFlowableSubscriber.onSuccess(MaybeToFlowable.java:70) at io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onSuccess(RxInstrumentedMaybeObserver.java:85) at io.reactivex.internal.operators.maybe.MaybeCreate$Emitter.onSuccess(MaybeCreate.java:73) at java.base/java.util.Optional.ifPresent(Optional.java:183) at io.micronaut.security.session.SessionAuthenticationFetcher.lambda$fetchAuthentication$0(SessionAuthenticationFetcher.java:53) at io.reactivex.internal.operators.maybe.MaybeCreate.subscribeActual(MaybeCreate.java:45) at io.reactivex.Maybe.subscribe(Maybe.java:4290) at io.micronaut.reactive.rxjava2.RxInstrumentedMaybe.subscribeActual(RxInstrumentedMaybe.java:53) at io.reactivex.Maybe.subscribe(Maybe.java:4290) at io.reactivex.internal.operators.maybe.MaybeToFlowable.subscribeActual(MaybeToFlowable.java:45) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onNext(FlowableFlatMap.java:163) at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:70) at io.reactivex.internal.operators.flowable.FlowableFromIterable$IteratorSubscription.slowPath(FlowableFromIterable.java:236) at io.reactivex.internal.operators.flowable.FlowableFromIterable$BaseRangeSubscription.request(FlowableFromIterable.java:124) at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onSubscribe(FlowableFlatMap.java:117) at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:55) at io.reactivex.internal.operators.flowable.FlowableFromIterable.subscribe(FlowableFromIterable.java:69) at io.reactivex.internal.operators.flowable.FlowableFromIterable.subscribeActual(FlowableFromIterable.java:47) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.internal.operators.flowable.FlowableFlatMap.subscribeActual(FlowableFlatMap.java:53) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe.subscribeActual(FlowableElementAtMaybe.java:36) at io.reactivex.Maybe.subscribe(Maybe.java:4290) at io.micronaut.reactive.rxjava2.RxInstrumentedMaybe.subscribeActual(RxInstrumentedMaybe.java:53) at io.reactivex.Maybe.subscribe(Maybe.java:4290) at io.reactivex.internal.operators.maybe.MaybeDoOnEvent.subscribeActual(MaybeDoOnEvent.java:39) at io.reactivex.Maybe.subscribe(Maybe.java:4290) at io.micronaut.reactive.rxjava2.RxInstrumentedMaybe.subscribeActual(RxInstrumentedMaybe.java:53) at io.reactivex.Maybe.subscribe(Maybe.java:4290) at io.reactivex.internal.operators.maybe.MaybeToFlowable.subscribeActual(MaybeToFlowable.java:45) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.internal.operators.flowable.FlowableFlatMap.subscribeActual(FlowableFlatMap.java:53) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty.subscribeActual(FlowableSwitchIfEmpty.java:32) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14868) at io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher.subscribe(WebMetricsPublisher.java:153) at io.reactivex.internal.operators.flowable.FlowableSwitchMap$SwitchMapSubscriber.onNext(FlowableSwitchMap.java:129) at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onNext(RxInstrumentedSubscriber.java:70) at io.micronaut.core.async.publisher.CompletableFuturePublisher$CompletableFutureSubscription.lambda$request$0(CompletableFuturePublisher.java:87) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883) at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251) at io.micronaut.core.async.publisher.CompletableFuturePublisher$CompletableFutureSubscription.request(CompletableFuturePublisher.java:81) at io.reactivex.internal.operators.flowable.FlowableSwitchMap$SwitchMapSubscriber.request(FlowableSwitchMap.java:162) at io.reactivex.internal.operators.flowable.FlowableSwitchMap$SwitchMapSubscriber.request(FlowableSwitchMap.java:162) at io.reactivex.internal.subscribers.BasicFuseableSubscriber.request(BasicFuseableSubscriber.java:153) at io.reactivex.internal.subscriptions.SubscriptionHelper.deferredSetOnce(SubscriptionHelper.java:202) at io.reactivex.internal.subscribers.StrictSubscriber.onSubscribe(StrictSubscriber.java:87) at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:55) at io.reactivex.internal.subscribers.BasicFuseableSubscriber.onSubscribe(BasicFuseableSubscriber.java:67) at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:55) at io.reactivex.internal.operators.flowable.FlowableSwitchMap$SwitchMapSubscriber.onSubscribe(FlowableSwitchMap.java:93) at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:55) at io.reactivex.internal.operators.flowable.FlowableSwitchMap$SwitchMapSubscriber.onSubscribe(FlowableSwitchMap.java:93) at io.micronaut.reactive.rxjava2.RxInstrumentedSubscriber.onSubscribe(RxInstrumentedSubscriber.java:55) at io.micronaut.core.async.publisher.CompletableFuturePublisher.subscribe(CompletableFuturePublisher.java:48) at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.internal.operators.flowable.FlowableSwitchMap.subscribeActual(FlowableSwitchMap.java:49) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.internal.operators.flowable.FlowableSwitchMap.subscribeActual(FlowableSwitchMap.java:49) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:37) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:58) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14868) at io.micronaut.http.context.ServerRequestTracingPublisher.lambda$subscribe$0(ServerRequestTracingPublisher.java:52) at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:68) at io.micronaut.http.context.ServerRequestTracingPublisher.subscribe(ServerRequestTracingPublisher.java:52) at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82) at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288) at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253) at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:79) at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:144) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedRunnable.run(InvocationInstrumenterWrappedRunnable.java:48) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.apache.kafka.common.config.ConfigException: Missing required configuration "bootstrap.servers" which has no default value. at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:477) at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:467) at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:108) at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:142) at org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:208) at org.apache.kafka.clients.admin.Admin.create(Admin.java:59) at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39) at org.akhq.modules.KafkaModule.getAdminClient(KafkaModule.java:100) at org.akhq.modules.AbstractKafkaWrapper.lambda$listTopics$1(AbstractKafkaWrapper.java:52) at org.akhq.utils.Logger.call(Logger.java:19) ... 156 common frames omitted 2020-03-20 11:10:06,097 WARN pGroup-1-2 org.akhq.log.access [Date: 2020-03-20T11:10:06.065044Z] [Duration: 32 ms] [Url: GET /favicon.ico/topic HTTP/1.1] [Status: 500] [Ip: 195.222.2.73] [Length: 445] [Port: 8080] 2020-03-20 11:10:54,235 INFO pGroup-1-2 org.akhq.log.access [Date: 2020-03-20T11:10:54.233194Z] [Duration: 1 ms] [Url: GET /health HTTP/1.1] [Status: 200] [Ip: 10.28.3.1] [Length: 215] [Port: 8080] 2020-03-20 11:11:54,233 INFO pGroup-1-2 org.akhq.log.access [Date: 2020-03-20T11:11:54.23139Z] [Duration: 1 ms] [Url: GET /health HTTP/1.1] [Status: 200] [Ip: 10.28.3.1] [Length: 215] [Port: 8080] 2020-03-20 11:12:12,208 INFO pGroup-1-2 org.akhq.log.access [Date: 2020-03-20T11:12:12.179394Z] [Duration: 29 ms] [Url: GET /login HTTP/1.1] [Status: 303] [Ip: 195.222.2.73] [Length: 0] [Port: 8080] 2020-03-20 11:12:24,232 INFO pGroup-1-2 org.akhq.log.access [Date: 2020-03-20T11:12:24.230903Z] [Duration: 1 ms] [Url: GET /health HTTP/1.1] [Status: 200] [Ip: 10.28.3.1] [Length: 215] [Port: 8080] 2020-03-20 11:12:59,371 INFO pGroup-1-2 org.akhq.log.access [Date: 2020-03-20T11:12:59.369286Z] [Duration: 1 ms] [Url: GET / HTTP/1.1] [Status: 301] [Ip: 178.124.154.7] [Length: 0] [Port: 8080]

iridian-ks commented 4 years ago

I believe I'm getting this too. I'm running Kafka and AKHQ in Kubernetes. Things work great initially and then after some time AKHQ stops responding. Simply redeploying all AKHQ containers gets things working again, but would like to know how to make this stop happening.

tchiotludo commented 4 years ago

must be related to this issue #209. I remove the metrics, it's available on dev version and maybe on a new release this week. Tell me if it works for you and reopen if not :)