tchiotludo / akhq

Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more...
https://akhq.io/
Apache License 2.0
3.41k stars 660 forks source link

Glue schema registry fails with SASL/SCRAM #1871

Closed mifonpe closed 3 months ago

mifonpe commented 4 months ago

Hi, first of all, thanks for supporting this amazing project!

We are moving from Confluent Cloud to AWS MSK and we wanted to verify feature parity when it comes to AKHQ. So far we have verified that we can connect to the MSK cluster without any issue.

However, when we add the schema registry definition for glue following this documentation, we receive an exception:

java.net.UnknownHostException: schema-registry
    at java.base/sun.nio.ch.NioSocketImpl.connect(Unknown Source)
    at java.base/java.net.Socket.connect(Unknown Source)
    at java.base/sun.net.NetworkClient.doConnect(Unknown Source)
    at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
    at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
    at java.base/sun.net.www.http.HttpClient.<init>(Unknown Source)
    at java.base/sun.net.www.http.HttpClient.New(Unknown Source)
    at java.base/sun.net.www.http.HttpClient.New(Unknown Source)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
    at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:312)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:408)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.getAllSubjects(RestService.java:1111)
    at io.confluent.kafka.schemaregistry.client.rest.RestService.getAllSubjects(RestService.java:1096)
    at org.akhq.repositories.SchemaRegistryRepository.all(SchemaRegistryRepository.java:104)
    at org.akhq.controllers.SchemaController.listAll(SchemaController.java:56)
    at org.akhq.controllers.$SchemaController$Definition$Exec.dispatch(Unknown Source)
    at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461)
    at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4276)
    at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:236)
    at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:488)
    at io.micronaut.http.server.RouteExecutor.lambda$callRoute$6(RouteExecutor.java:465)
    at io.micronaut.core.execution.ExecutionFlow.lambda$async$1(ExecutionFlow.java:87)
    at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
    at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:193)
    at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
    at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
    at io.micrometer.core.instrument.Timer.lambda$wrap$0(Timer.java:193)
    at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)

Our configuration file looks somewhat like this (removing sensitive values):

akhq:
  connections:
    msk:
      properties:
        bootstrap.servers: "bootstrap-server-dns-name:9096"
        security.protocol: SASL_SSL
        sasl.mechanism: SCRAM-SHA-512
        sasl.jaas.config: org.apache.kafka.common.security.scram.ScramLoginModule required username="<username>" password="<password>";
      schema-registry:
        url: "http://schema-registry:8085"
        type: "glue"
        glueSchemaRegistryName: "<name-of-the-registry>"
        awsRegion: "<aws-region>"

Besides we are passing AWS Credentials (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY ) to the container where AKHQ is running, as the documentation mentions it relies on the DefaultCredentialsProvider. I have verified that the values are available as environment variables.

Extra: I have verified in cloudTrail and I have not seen any calls from the user which owns the secret access key/ access key pair so I assume it is not a permissions error. Plus I gave the user full access to verify and it still fails.

Let me know if you need some more information, and thanks in advance for giving it a look 🙂

AlexisSouquiere commented 4 months ago

For me the error is self-explanatory. UnknownHostException means that the container cannot find the IP address for the hostname (the real hostname instead of http://schema-registry) you are giving in the configuration. So it seems normal that you don't see any traces in cloudtrail because the request doesn't even reach the registry. You should check if there is any network rules that are preventing access from where you deployed AKHQ.