tchiotludo / akhq

Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more...
https://akhq.io/
Apache License 2.0
3.41k stars 659 forks source link

Kafkahq with sasl/kerberos (gssapi) based kafka cluster #176

Closed fshabir closed 4 years ago

fshabir commented 4 years ago

Hello I am trying to try out kafkahq with a kerberos backed Kafka cluster. Below are setup details

Application configuration is:

kafkahq.connections: key: "test-cluster-2.4" properties: bootstrap.servers: kafkatest1:9092 security.protocol: SASL_PLAINTEXT group.id: kafkahq-consumer sasl.kerberos.service.name: kafkains sasl.jaas.config: 'com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="PATH_TO_KEYTAB" principal="PRINCIPAL NAME";' micronaut.server.port: 7070

When I start kafkahq, it shows following:

2020-01-06 02:58:23,904 ERROR pGroup-1-2 o.k.c.ErrorController Error for List topics java.lang.RuntimeException: Error for List topics at org.kafkahq.utils.Logger.call(Logger.java:26) at org.kafkahq.modules.AbstractKafkaWrapper.listTopics(AbstractKafkaWrapper.java:50) at org.kafkahq.modules.$KafkaWrapperRequestScopeDefinition$$exec2.invokeInternal(Unknown Source) at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:145) at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:60) at org.kafkahq.modules.$KafkaWrapperRequestScopeDefinition$Intercepted.listTopics(Unknown Source) at org.kafkahq.repositories.TopicRepository.all(TopicRepository.java:70) at org.kafkahq.repositories.TopicRepository.list(TopicRepository.java:62) at org.kafkahq.controllers.TopicController.list(TopicController.java:101) at org.kafkahq.controllers.$TopicControllerDefinition$$exec1.invokeInternal(Unknown Source) at io.micronaut.context.AbstractExecutableMethod.invoke(AbstractExecutableMethod.java:145) at io.micronaut.context.DefaultBeanContext$BeanExecutionHandle.invoke(DefaultBeanContext.java:2844) at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:294) at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:122) at io.micronaut.http.server.netty.RoutingInBoundHandler.lambda$buildResultEmitter$17(RoutingInBoundHandler.java:1400) at io.reactivex.internal.operators.flowable.FlowableCreate.subscribeActual(FlowableCreate.java:71) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:37) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty.subscribeActual(FlowableSwitchIfEmpty.java:32) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.internal.operators.flowable.FlowableSwitchMap.subscribeActual(FlowableSwitchMap.java:49) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.reactivex.internal.operators.flowable.FlowableDefer.subscribeActual(FlowableDefer.java:42) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FlowableSwitchIfEmpty.java:71) at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52) at io.micronaut.http.context.ServerRequestContext.lambda$instrument$0(ServerRequestContext.java:68) at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.onComplete(InstrumentedSubscriber.java:112) at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drainLoop(FlowableFlatMap.java:426) at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drain(FlowableFlatMap.java:366) at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onComplete(FlowableFlatMap.java:338) at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52) at io.micronaut.http.context.ServerRequestContext.lambda$instrument$0(ServerRequestContext.java:68) at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.onComplete(InstrumentedSubscriber.java:112) at io.reactivex.internal.operators.maybe.MaybeToFlowable$MaybeToFlowableSubscriber.onComplete(MaybeToFlowable.java:80) at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52) at io.micronaut.http.context.ServerRequestContext.lambda$instrument$0(ServerRequestContext.java:68) at io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onComplete(RxInstrumentedMaybeObserver.java:92) at io.reactivex.internal.operators.maybe.MaybeDoOnEvent$DoOnEventMaybeObserver.onComplete(MaybeDoOnEvent.java:115) at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52) at io.micronaut.http.context.ServerRequestContext.lambda$instrument$0(ServerRequestContext.java:68) at io.micronaut.reactive.rxjava2.RxInstrumentedMaybeObserver.onComplete(RxInstrumentedMaybeObserver.java:92) at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe$ElementAtSubscriber.onComplete(FlowableElementAtMaybe.java:102) at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52) at io.micronaut.http.context.ServerRequestContext.lambda$instrument$0(ServerRequestContext.java:68) at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.onComplete(InstrumentedSubscriber.java:112) at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drainLoop(FlowableFlatMap.java:426) at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drain(FlowableFlatMap.java:366) at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onComplete(FlowableFlatMap.java:338) at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52) at io.micronaut.http.context.ServerRequestContext.lambda$instrument$0(ServerRequestContext.java:68) at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.onComplete(InstrumentedSubscriber.java:112) at io.reactivex.internal.operators.flowable.FlowableFromIterable$IteratorSubscription.slowPath(FlowableFromIterable.java:255) at io.reactivex.internal.operators.flowable.FlowableFromIterable$BaseRangeSubscription.request(FlowableFromIterable.java:124) at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onSubscribe(FlowableFlatMap.java:117) at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.onSubscribe(InstrumentedSubscriber.java:75) at io.reactivex.internal.operators.flowable.FlowableFromIterable.subscribe(FlowableFromIterable.java:69) at io.reactivex.internal.operators.flowable.FlowableFromIterable.subscribeActual(FlowableFromIterable.java:47) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.internal.operators.flowable.FlowableFlatMap.subscribeActual(FlowableFlatMap.java:53) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe.subscribeActual(FlowableElementAtMaybe.java:36) at io.reactivex.Maybe.subscribe(Maybe.java:4290) at io.micronaut.reactive.rxjava2.RxInstrumentedMaybe.subscribeActual(RxInstrumentedMaybe.java:64) at io.reactivex.Maybe.subscribe(Maybe.java:4290) at io.reactivex.internal.operators.maybe.MaybeDoOnEvent.subscribeActual(MaybeDoOnEvent.java:39) at io.reactivex.Maybe.subscribe(Maybe.java:4290) at io.micronaut.reactive.rxjava2.RxInstrumentedMaybe.subscribeActual(RxInstrumentedMaybe.java:64) at io.reactivex.Maybe.subscribe(Maybe.java:4290) at io.reactivex.internal.operators.maybe.MaybeToFlowable.subscribeActual(MaybeToFlowable.java:45) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.internal.operators.flowable.FlowableFlatMap.subscribeActual(FlowableFlatMap.java:53) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.internal.operators.flowable.FlowableSwitchIfEmpty.subscribeActual(FlowableSwitchIfEmpty.java:32) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14868) at io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher.subscribe(WebMetricsPublisher.java:122) at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.internal.operators.flowable.FlowableSwitchMap.subscribeActual(FlowableSwitchMap.java:49) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:37) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14868) at io.micronaut.http.context.ServerRequestTracingPublisher.lambda$subscribe$0(ServerRequestTracingPublisher.java:52) at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:52) at io.micronaut.http.context.ServerRequestTracingPublisher.subscribe(ServerRequestTracingPublisher.java:52) at io.reactivex.internal.operators.flowable.FlowableFromPublisher.subscribeActual(FlowableFromPublisher.java:29) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.micronaut.reactive.rxjava2.RxInstrumentedFlowable.subscribeActual(RxInstrumentedFlowable.java:68) at io.reactivex.Flowable.subscribe(Flowable.java:14918) at io.reactivex.Flowable.subscribe(Flowable.java:14865) at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82) at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:288) at io.reactivex.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:253) at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:79) at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:144) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.lang.NullPointerException: null at org.kafkahq.modules.KafkaModule.lambda$getDefaultsProperties$3(KafkaModule.java:64) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1654) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) at org.kafkahq.modules.KafkaModule.getDefaultsProperties(KafkaModule.java:63) at org.kafkahq.modules.KafkaModule.getAdminProperties(KafkaModule.java:89) at org.kafkahq.modules.KafkaModule.getAdminClient(KafkaModule.java:98) at org.kafkahq.modules.AbstractKafkaWrapper.lambda$listTopics$1(AbstractKafkaWrapper.java:51) at org.kafkahq.utils.Logger.call(Logger.java:19) ... 145 common frames omitted

The error message itself doesn't seem to be of much help, so I was wondering if KafkaHQ supports integration with Kerberos (GSSAPI) based authentication scheme for SASL_PLAINTEXT mode? If, yes then how to specify the JAAS and KRB5 configs?

tchiotludo commented 4 years ago

Hello,

Can you provide me the whole yaml please ? I suspected indentation issue that break the application. The null pointer exception was on part of the config that you don't provide on the issue. Also the configuration show me a kafka.connections with a key inside, there is no key on the configuration, look at application.example.yml at the root of the repo for a full example

fshabir commented 4 years ago

hi, changed yaml and now application.yml looks like this:

kafkahq.connections:
  test-cluster-2.4:
    properties:
      bootstrap.servers: BROKER_HOST:9092
      security.protocol: SASL_PLAINTEXT
      group.id: kafkahq-consumer
      sasl.kerberos.service.name: kafkaservice
      sasl.jaas.config: com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="/etc/keytabs/mykeytab.keytab" principal="principal@keberos_realm";
micronaut.server.port: 7070
tchiotludo commented 4 years ago

try with that :

kafkahq:
  connections: 
    test-cluster-2.4:
      properties:
        bootstrap.servers: BROKER_HOST:9092
        security.protocol: SASL_PLAINTEXT
        group.id: kafkahq-consumer
        sasl.kerberos.service.name: kafkaservice
        sasl.jaas.config: com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="/etc/keytabs/mykeytab.keytab" principal="principal@keberos_realm";
micronaut:
  server:
    port: 7070
fshabir commented 4 years ago

Changed the configs as you provided, now strangely stuck at this:

Caused by: org.apache.kafka.common.config.ConfigException: Missing required configuration "bootstrap.servers" which has no default value.
        at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:476)
        at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:466)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:108)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:142)
        at org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:196)
        at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:55)
        at org.kafkahq.modules.KafkaModule.getAdminClient(KafkaModule.java:98)
        at org.kafkahq.modules.AbstractKafkaWrapper.lambda$listTopics$1(AbstractKafkaWrapper.java:51)
        at org.kafkahq.utils.Logger.call(Logger.java:19)
        ... 145 common frames omitted

Although bootstrap.servers is provided in the config

tchiotludo commented 4 years ago

forgot the quote on bootstrap server I think since : is a reserved char on yaml

kafkahq:
  connections: 
    test-cluster-2.4:
      properties:
        bootstrap.servers: "BROKER_HOST:9092"
        security.protocol: SASL_PLAINTEXT
        group.id: kafkahq-consumer
        sasl.kerberos.service.name: kafkaservice
        sasl.jaas.config: com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="/etc/keytabs/mykeytab.keytab" principal="principal@keberos_realm";
micronaut:
  server:
    port: 7070
fshabir commented 4 years ago

Already tried it with the quotes around bootstrap.servers with no success. Now my yaml looks like this:

kafkahq:
  connections: 
    test-cluster-2:
      properties:
        bootstrap.servers: "BROKER_HOST:9092"
        security.protocol: SASL_PLAINTEXT
        group.id: kafkahq-consumer
        sasl.kerberos.service.name: kafkaservice
        sasl.jaas.config: com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="/etc/keytabs/mykeytab.keytab" principal="principal@keberos_realm";
micronaut:
  server:
    port: 7070
tchiotludo commented 4 years ago

add this to your configuration files :

endpoints:
  env:
    enabled: true
    sensitive: false

and go to http://localhost:7070/env, it will show the configuration read by KafkaHQ and allow you to debug it

fshabir commented 4 years ago

I tried it with debug mode but it looks okay:

{
  "packages": [
    "org.kafkahq"
  ],
  "propertySources": [
    {
      "name": "application",
      "order": -300,
      "convention": "JAVA_PROPERTIES",
      "properties": {
        "micronaut.application.name": "kafkahq",
        "micronaut.io.watch.paths": "src/main",
        "micronaut.io.watch.restart": false,
        "micronaut.router.static-resources.static.paths": "classpath:static",
        "micronaut.router.static-resources.static.mapping": "${kafkahq.server.base-path:}/static/**",
        "micronaut.security.enabled": true,
        "micronaut.security.endpoints.login.enabled": true,
        "micronaut.security.endpoints.login.path": "${kafkahq.server.base-path:}/login",
        "micronaut.security.endpoints.logout.enabled": true,
        "micronaut.security.endpoints.logout.path": "${kafkahq.server.base-path:}/logout",
        "micronaut.security.endpoints.logout.get-allowed": true,
        "micronaut.security.session.enabled": true,
        "micronaut.security.session.login-success-target-url": "${kafkahq.server.base-path:}/",
        "micronaut.security.session.logout-target-url": "${kafkahq.server.base-path:}/",
        "micronaut.security.session.forbidden-target-url": "${kafkahq.server.base-path:}/login/forbidden",
        "micronaut.security.session.unauthorized-target-url": "${kafkahq.server.base-path:}/login/unauthorized",
        "micronaut.security.session.login-failure-target-url": "${kafkahq.server.base-path:}/login/failed",
        "micronaut.security.intercept-url-map": [
          {
            "pattern": "${kafkahq.server.base-path:}/static/**",
            "access": "isAnonymous()"
          }
        ],
        "micronaut.caches.kafka-wrapper.record-stats": true,
        "micronaut.caches.kafka-wrapper.expire-after-write": "0s",
        "endpoints.all.path": "${kafkahq.server.base-path:}",
        "endpoints.health.enabled": true,
        "endpoints.health.sensitive": false,
        "endpoints.health.details-visible": "anonymous",
        "endpoints.info.enabled": true,
        "endpoints.info.sensitive": false,
        "endpoints.metrics.enabled": true,
        "endpoints.metrics.sensitive": false,
        "endpoints.metrics.export.prometheus.enabled": true,
        "endpoints.metrics.export.prometheus.step": "PT1M",
        "endpoints.metrics.export.prometheus.descriptions": true,
        "endpoints.prometheus.enabled": true,
        "endpoints.prometheus.sensitive": false,
        "endpoints.caches.enabled": true,
        "endpoints.caches.sensitive": false,
        "jackson.module-scan": false,
        "kafkahq.server.access-log.enabled": true,
        "kafkahq.server.access-log.name": "org.kafkahq.log.access",
        "kafkahq.server.access-log.format": "[Date: {}] [Duration: {} ms] [Url: {} {} {}] [Status: {}] [Ip: {}] [Length: {}] [Port: {}]",
        "kafkahq.clients-defaults.consumer.properties.max.poll.records": 50,
        "kafkahq.clients-defaults.consumer.properties.isolation.level": "read_committed",
        "kafkahq.clients-defaults.consumer.properties.group.id": "KafkaHQ",
        "kafkahq.clients-defaults.consumer.properties.enable.auto.commit": "false",
        "kafkahq.clients-defaults.consumer.properties.default.api.timeout.ms": 15000,
        "kafkahq.pagination.page-size": 25,
        "kafkahq.pagination.threads": 16,
        "kafkahq.topic.default-view": "HIDE_INTERNAL",
        "kafkahq.topic.replication": 1,
        "kafkahq.topic.retention": 86400000,
        "kafkahq.topic.partition": 1,
        "kafkahq.topic.internal-regexps": [
          "^_.*$",
          "^.*_schemas$",
          "^.*connect-config$",
          "^.*connect-offsets$1",
          "^.*connect-status$"
        ],
        "kafkahq.topic.stream-regexps": [
          "^.*-changelog$",
          "^.*-repartition$",
          "^.*-rekey$"
        ],
        "kafkahq.topic.skip-consumer-groups": false,
        "kafkahq.topic-data.sort": "OLDEST",
        "kafkahq.topic-data.size": 50,
        "kafkahq.topic-data.poll-timeout": 1000,
        "kafkahq.security.default-groups": [
          "admin"
        ],
        "kafkahq.security.groups.admin.roles": [
          "topic/read",
          "topic/insert",
          "topic/delete",
          "topic/config/update",
          "node/read",
          "node/config/update",
          "topic/data/read",
          "topic/data/insert",
          "topic/data/delete",
          "group/read",
          "group/delete",
          "group/offsets/update",
          "registry/read",
          "registry/insert",
          "registry/update",
          "registry/delete",
          "registry/version/delete",
          "acls/read",
          "connect/read",
          "connect/insert",
          "connect/update",
          "connect/delete",
          "connect/state/update"
        ],
        "kafkahq.security.groups.reader.roles": [
          "topic/read",
          "node/read",
          "topic/data/read",
          "group/read",
          "registry/read",
          "acls/read",
          "connect/read"
        ]
      }
    },
    {
      "name": "env",
      "order": -200,
      "convention": "ENVIRONMENT_VARIABLE",
      "properties": {
        "PATH": "/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/myhome/.local/bin:/home/myhome/bin:/home/myhome/jdk-11/bin",
        "HISTCONTROL": "ignoredups",
        "HISTSIZE": "5000",
        "JAVA_HOME": "/home/myhome/jdk-11",
        "TERM": "screen",
        "LANG": "en_US.UTF-8",
        "JRE_HOME": "/home/myhome/jdk-11/bin",
        "MAIL": "/var/spool/mail/myusername",
        "LOGNAME": "myusername",
        "JDK_HOME": "/home/myhome/jdk-11",
        "PWD": "/home/myhome/kafkahq",
        "HISTTIMEFORMAT": "%m/%d %H:%M:%S  ",
        "_": "/home/shabirfurhan01/jdk-11/bin/java",
        "LESSOPEN": "||/usr/bin/lesspipe.sh %s",
        "SHELL": "/bin/bash",
        "QT_GRAPHICSSYSTEM_CHECKED": "1",
        "QTINC": "/usr/lib64/qt-3.3/include",
        "SSH_TTY": "/dev/pts/4",
        "SSH_CLIENT": "xxx.xxx.xxx.xxx 22",
        "USER": "myusername",
        "SSH_CONNECTION": "xxx.xxx.xxx.xxx",
        "HOSTNAME": "BROKER_HOST",
        "QTDIR": "/usr/lib64/qt-3.3",
        "LS_COLORS": "rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:",
        "QTLIB": "/usr/lib64/qt-3.3/lib",
        "SHLVL": "2",
        "HOME": "/home/myhome"
      }
    },
    {
      "name": "system",
      "order": -100,
      "convention": "JAVA_PROPERTIES",
      "properties": {
        "awt.toolkit": "sun.awt.X11.XToolkit",
        "java.specification.version": "11",
        "sun.jnu.encoding": "UTF-8",
        "java.class.path": "kafkahq.jar",
        "java.vm.vendor": "Oracle Corporation",
        "sun.arch.data.model": "64",
        "micronaut.config.files": "/home/myhome/kafkahq/application.yml",
        "java.vendor.url": "http://java.oracle.com/",
        "user.timezone": "UTC",
        "os.name": "Linux",
        "java.vm.specification.version": "11",
        "sun.java.launcher": "SUN_STANDARD",
        "user.country": "US",
        "sun.boot.library.path": "/home/myhome/jdk-11/lib",
        "sun.java.command": "kafkahq.jar",
        "jdk.debug": "release",
        "sun.cpu.endian": "little",
        "user.home": "/home/myhome",
        "user.language": "en",
        "java.specification.vendor": "Oracle Corporation",
        "java.version.date": "2018-09-25",
        "java.home": "/home/myhome/jdk-11",
        "file.separator": "/",
        "java.vm.compressedOopsMode": "32-bit",
        "line.separator": "\n",
        "java.specification.name": "Java Platform API Specification",
        "java.vm.specification.vendor": "Oracle Corporation",
        "java.awt.graphicsenv": "sun.awt.X11GraphicsEnvironment",
        "sun.management.compiler": "HotSpot 64-Bit Tiered Compilers",
        "java.runtime.version": "11+28",
        "user.name": "myusername",
        "path.separator": ":",
        "os.version": "4.15.0-42-generic",
        "java.runtime.name": "OpenJDK Runtime Environment",
        "file.encoding": "UTF-8",
        "java.vm.name": "OpenJDK 64-Bit Server VM",
        "java.vendor.version": "18.9",
        "micronaut.classloader.logging": "true",
        "java.vendor.url.bug": "http://bugreport.java.com/bugreport/",
        "java.io.tmpdir": "/tmp",
        "java.version": "11",
        "user.dir": "/home/myhome/kafkahq",
        "os.arch": "amd64",
        "java.vm.specification.name": "Java Virtual Machine Specification",
        "java.awt.printerjob": "sun.print.PSPrinterJob",
        "sun.os.patch.level": "unknown",
        "java.library.path": "/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib",
        "java.vendor": "Oracle Corporation",
        "java.vm.info": "mixed mode",
        "java.vm.version": "11+28",
        "sun.io.unicode.encoding": "UnicodeLittle",
        "java.class.version": "55.0"
      }
    },
    {
      "name": "cli",
      "order": 0,
      "convention": "JAVA_PROPERTIES"
    },
    {
      "name": "/home/myhome/kafkahq/application.yml",
      "order": 0,
      "convention": "JAVA_PROPERTIES",
      "properties": {
        "kafkahq.connections.test-cluster-2.properties.bootstrap.servers": "BROKER_HOST:9092",
        "kafkahq.connections.test-cluster-2.properties.security.protocol": "SASL_PLAINTEXT",
        "kafkahq.connections.test-cluster-2.properties.group.id": "kafkahq-consumer",
        "kafkahq.connections.test-cluster-2.properties.sasl.kerberos.service.name": "kafkaservice",
        "kafkahq.connections.test-cluster-2.properties.sasl.jaas.config": "com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab=\"/etc/keytabs/mykeytab.keytab\" principal=\"principal@keberos_realm\";",
        "micronaut.server.host": "BROKER_HOST",
        "micronaut.server.port": 7070,
        "endpoints.env.enabled": true,
        "endpoints.env.sensitive": false
      }
    }
  ]
}
fshabir commented 4 years ago

hello @tchiotludo any rough idea, what might be wrong with my configuration, by looking at the output of my config from debug endpoint?

tchiotludo commented 4 years ago

Hi @fshabir,

Sorry forgot to respond you ! To be honest, I don't see any trouble on your configuration. The debug output is telling me that your configuration files seems to be ok.

What is the url you try to see ? I suppose : http://localhost:7070/test-cluster-2/topic

Can you send me the error in attached files (to keep indentation) ? Thanks

apellegr06 commented 4 years ago

Hi,

It works for me, here is my connection configuration :

connections: MY_KAFKA: properties: bootstrap.servers: "hostname01:6668,hostname02:6668" security.protocol: SASL_PLAINTEXT sasl.mechanism: GSSAPI sasl.jaas.config: com.sun.security.auth.module.Krb5LoginModule required renewTGT=false doNotPrompt=true useKeyTab=true keyTab="/tmp/kafka.keytab" storeKey=true useTicketCache=false serviceName="kafka" principal="kafka@MY_DOMAIN.FR";