search

tchiotludo / akhq

Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more...
https://akhq.io/
Apache License 2.0
3.42k stars 661 forks source link

Cannot connect to Kafka Broker using SASL with OAUTHBEARER #969

Open jheinitz opened 2 years ago

jheinitz commented 2 years ago

Hello,

I'm trying to connect to my Kafka Broker using SASL. I'm using the docker version 0.20.0. I can successfully connect to the broker when running

kafka-topics --bootstrap-server broker:9094 --command-config command_sasl.properties --list

The command_sasl.properties lokks like this:

security.protocol=SASL_SSL ssl.key.password=<pass> ssl.keystore.location=keystore.jks ssl.keystore.password=<pass> ssl.truststore.location=truststore.jks ssl.truststore.password=<pass> sasl.mechanism=OAUTHBEARER sasl.login.callback.handler.class=io.confluent.kafka.clients.plugins.auth.token.TokenUserLoginCallbackHandler sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \ username="username" \ password="password" \ metadataServerUrls="https://broker:8090" ;

My appllication.yml looks like this (sorry for the missing indents):

akhq:
  connections:
    my-sasl:
      properties:
        bootstrap.servers: "broker:9094"
        security.protocol: SASL_SSL
        ssl.key.password: pass
        ssl.keystore.location: /akhq/keystore.p12
        ssl.keystore.password: pass
        ssl.keystore.type: PKCS12
        ssl.truststore.location: /akhq/truststore.jks
        ssl.truststore.password: pass
        sasl.mechanism: OAUTHBEARER
        sasl.login.callback.handler.class: io.confluent.kafka.clients.plugins.auth.token.TokenUserLoginCallbackHandler
        sasl.jaas.config: org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required username="username" password="password" metadataServerUrls="https://broker:8090" ;

But when I try to connect to the cluster, I get this error:

akhq_1 | 2021-12-23 15:49:14,712 ERROR r-thread-9 o.a.c.ErrorController Invalid value io.confluent.kafka.clients.plugins.auth.token.TokenUserLoginCallbackHandler for configuration sasl.login.callback.handler.class: Class io.confluent.kafka.clients.plugins.auth.token.TokenUserLoginCallbackHandler could not be found.

Any ideas?

Thanks and best regards

Jens

tchiotludo commented 2 years ago

There is a missing deps on akhq side, was added, you can test on dev branch in 20 minutes

jheinitz commented 2 years ago

Hi! That was quick. Thanks a lot. Will be able to test not before beginning of January. Will give feedback then.

jheinitz commented 2 years ago

Hello!

Happy New Year to you and all others reading this.

I found some time to test the dev docker image to see if I could connect to my Kafka Cluster through SASL, but I still fail to connect. But this time the error is different from the one I saw before. I'm using the docker image tag

tchiotludo/akhq:dev (id:8ffb081b4d53)

This what I get when I try to connect:

akhq_1       | 2022-01-03 10:49:44,434 ERROR r-thread-3 o.a.c.ErrorController      Failed to create new KafkaAdminClient
akhq_1       | org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
akhq_1       |  at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:527)
akhq_1       |  at org.apache.kafka.clients.admin.Admin.create(Admin.java:133)
akhq_1       |  at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39)
akhq_1       |  at org.akhq.modules.KafkaModule.getAdminClient(KafkaModule.java:98)
akhq_1       |  at org.akhq.modules.AbstractKafkaWrapper.describeCluster(AbstractKafkaWrapper.java:35)
akhq_1       |  at org.akhq.modules.$KafkaWrapperRequestScope$Definition$Exec.dispatch(Unknown Source)
akhq_1       |  at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:351)
akhq_1       |  at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:128)
akhq_1       |  at org.akhq.modules.$KafkaWrapperRequestScope$Definition$Intercepted.describeCluster(Unknown Source)
akhq_1       |  at org.akhq.repositories.ClusterRepository.get(ClusterRepository.java:16)
akhq_1       |  at org.akhq.controllers.NodeController.list(NodeController.java:41)
akhq_1       |  at org.akhq.controllers.$NodeController$Definition$Exec.dispatch(Unknown Source)
akhq_1       |  at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:351)
akhq_1       |  at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:583)
akhq_1       |  at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:303)
akhq_1       |  at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111)
akhq_1       |  at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103)
akhq_1       |  at io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:656)
akhq_1       |  at reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49)
akhq_1       |  at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)
akhq_1       |  at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194)
akhq_1       |  at io.micronaut.reactive.reactor.instrument.ReactorInstrumentation.lambda$null$0(ReactorInstrumentation.java:62)
akhq_1       |  at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
akhq_1       |  at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
akhq_1       |  at io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedCallable.call(InvocationInstrumenterWrappedCallable.java:53)
akhq_1       |  at io.micrometer.core.instrument.composite.CompositeTimer.recordCallable(CompositeTimer.java:68)
akhq_1       |  at io.micrometer.core.instrument.Timer.lambda$wrap$1(Timer.java:171)
akhq_1       |  at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
akhq_1       |  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
akhq_1       |  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
akhq_1       |  at java.base/java.lang.Thread.run(Unknown Source)
akhq_1       | Caused by: org.apache.kafka.common.KafkaException: java.lang.NoSuchMethodError: 'org.apache.kafka.common.security.ssl.DefaultSslEngineFactory org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.castOrThrow(org.apache.kafka.common.security.auth.SslEngineFactory)'
akhq_1       |  at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:184)
akhq_1       |  at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
akhq_1       |  at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
akhq_1       |  at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:104)
akhq_1       |  at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:501)
akhq_1       |  ... 30 common frames omitted
akhq_1       | Caused by: java.lang.NoSuchMethodError: 'org.apache.kafka.common.security.ssl.DefaultSslEngineFactory org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.castOrThrow(org.apache.kafka.common.security.auth.SslEngineFactory)'
akhq_1       |  at io.confluent.security.auth.client.rest.RestClient.createSslSocketFactory(RestClient.java:204)
akhq_1       |  at io.confluent.security.auth.client.rest.RestClient.<init>(RestClient.java:130)
akhq_1       |  at io.confluent.security.auth.client.rest.RestClient.<init>(RestClient.java:100)
akhq_1       |  at io.confluent.kafka.clients.plugins.auth.token.TokenUserLoginCallbackHandler.configure(TokenUserLoginCallbackHandler.java:67)
akhq_1       |  at io.confluent.kafka.clients.plugins.auth.token.AbstractTokenLoginCallbackHandler.configure(AbstractTokenLoginCallbackHandler.java:86)
akhq_1       |  at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:60)
akhq_1       |  at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:105)
akhq_1       |  at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:170)
akhq_1       |  ... 34 common frames omitted

Could it be that there is another dependency missing?

Thanks and kind regards

Jens

jheinitz commented 2 years ago

Hi @tchiotludo ,

is there a way for me to re-open this issue?

Cheers

Jens

tchiotludo commented 2 years ago

@jheinitz I don't think I can resolve this one : The stacktrace goes here and there is no occurence of castOrThrow

The code on the confluent package added : 

    private SSLSocketFactory createSslSocketFactory(Map<String, ?> sslConfigs) {
        SslFactory sslFactory = new SslFactory(Mode.CLIENT);
        sslFactory.configure(sslConfigs);
        return DefaultSslEngineFactory.castOrThrow(sslFactory.sslEngineFactory()).sslContext().getSocketFactory();
    }

where castOrThrow is called and don't exist anywhere...

The only mention is here with this stacktrace I found: https://github.com/kafka-ops/julie/issues/241

Since the code is not open source for this package I don't even know where you can report the issue ... but we can't do anything on that for now

jheinitz commented 2 years ago

Bon jour!

Thanks for looking into it. When I go through the coments in kafka-ops/julie#241 I can see that purpon is using all jars from Confluent for Kafka Julie to avoid problems with RBAC. I think that is hapening due to the fact that RBAC is pure Confluent stuff and not available in the basic kafka packages. Am I right?

Would it be an option to provide a test image where you do something similar using the jars from confluent?

Thx and best regards

Jens

PS: You wrote that the code is not open source. Do you mean the confluent code?

tchiotludo commented 2 years ago

Not really sure for the julie issues, I don't know the project myself. Just found some people with the same issues but clearly the code (from confluent package kafka-client-plugins) couldn't work as is. I'll try to find quickly with an older version of Kafka and don't find the method requested by this package.

PS: You wrote that the code is not open source. Do you mean the confluent code? yes, there is no public github as I see

To be honest, I don't even know this auth since I don't use, you have a sample application (producer / consumer / stream) in java that works with this libs ?

jheinitz commented 2 years ago

Hello @tchiotludo !

Not really sure for the julie issues, I don't know the project myself.

I just confirmed with my colleague tha we successfully connected to the same Kafka Cluster using Julie.

To be honest, I don't even know this auth since I don't use, you have a sample application (producer / consumer / stream) in java that works with this libs ?

We are currently working on it. As soon as I have something to show, I will let you know.

Thanks and kind regards

Jens