search

provectus / kafka-ui

Open-Source Web UI for Apache Kafka Management
Apache License 2.0
9.81k stars 1.19k forks source link

Failing to connect to Confluent Cloud Kafka #936

Closed harlev closed 2 years ago

harlev commented 3 years ago

Sorry for the typo in https://github.com/provectus/kafka-ui/issues/920. After I fix this and use these settings

I get org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient

In detail:

kafka-ui  | 16:04:57.390 [main] INFO  org.springframework.core.KotlinDetector - Kotlin reflection implementation not found at runtime, related features won't be available.
kafka-ui  |
kafka-ui  |   .   ____          _            __ _ _
kafka-ui  |  /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
kafka-ui  | ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
kafka-ui  |  \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
kafka-ui  |   '  |____| .__|_| |_|_| |_\__, | / / / /
kafka-ui  |  =========|_|==============|___/=/_/_/_/
kafka-ui  |  :: Spring Boot ::        (v2.2.4.RELEASE)
kafka-ui  |
kafka-ui  | 16:04:57.981 [main] INFO  com.provectus.kafka.ui.KafkaUiApplication - Starting KafkaUiApplication on b64d97c898ac with PID 1 (/kafka-ui-api.jar started by root in /)
kafka-ui  | 16:04:57.982 [main] DEBUG com.provectus.kafka.ui.KafkaUiApplication - Running with Spring Boot v2.2.4.RELEASE, Spring v5.2.3.RELEASE
kafka-ui  | 16:04:57.982 [main] INFO  com.provectus.kafka.ui.KafkaUiApplication - No active profile set, falling back to default profiles: default
kafka-ui  | 16:04:58.253 [background-preinit] WARN  org.springframework.http.converter.json.Jackson2ObjectMapperBuilder - For Jackson Kotlin classes support please add "com.fasterxml.jackson.module:jackson-module-kotlin" to the classpath
kafka-ui  | SLF4J: Class path contains multiple SLF4J bindings.
kafka-ui  | SLF4J: Found binding in [jar:file:/kafka-ui-api.jar!/BOOT-INF/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
kafka-ui  | SLF4J: Found binding in [jar:file:/kafka-ui-api.jar!/BOOT-INF/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
kafka-ui  | SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
kafka-ui  | SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
kafka-ui  | log4j:WARN No appenders could be found for logger (reactor.util.Loggers$LoggerFactory).
kafka-ui  | log4j:WARN Please initialize the log4j system properly.
kafka-ui  | log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
kafka-ui  | 16:04:59.357 [main] WARN  org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JodaDateTimeJacksonConfiguration - Auto-configuration of Jackson's Joda-Time integration is deprecated in favor of using java.time (JSR-310).
kafka-ui  | 16:04:59.388 [main] INFO  com.provectus.kafka.ui.serde.DeserializationService - Using SchemaRegistryAwareRecordSerDe for cluster 'azureDev'
kafka-ui  | 16:04:59.820 [main] INFO  org.springframework.boot.autoconfigure.security.reactive.ReactiveUserDetailsServiceAutoConfiguration -
kafka-ui  |
kafka-ui  | Using generated security password: *************
kafka-ui  |
kafka-ui  | 16:05:00.341 [main] INFO  org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler - Initializing ExecutorService 'taskScheduler'
kafka-ui  | 16:05:00.447 [parallel-1] DEBUG com.provectus.kafka.ui.service.MetricsUpdateService - Start getting metrics for kafkaCluster: azureDev
kafka-ui  | 16:05:00.620 [main] INFO  org.springframework.boot.web.embedded.netty.NettyWebServer - Netty started on port(s): 8080
kafka-ui  | 16:05:00.624 [main] INFO  com.provectus.kafka.ui.KafkaUiApplication - Started KafkaUiApplication in 3.173 seconds (JVM running for 4.416)
kafka-ui  | 16:05:00.810 [parallel-1] ERROR com.provectus.kafka.ui.service.KafkaService - Failed to collect cluster azureDev info
kafka-ui  | org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
kafka-ui  |     at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:540) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  |     at org.apache.kafka.clients.admin.Admin.create(Admin.java:134) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  |     at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  |     at com.provectus.kafka.ui.service.AdminClientServiceImpl.lambda$createAdminClient$2(AdminClientServiceImpl.java:41) ~[classes!/:?]
kafka-ui  |     at reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:85) ~[reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  |     at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:126) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  |     at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  |     at reactor.core.publisher.Mono.subscribe(Mono.java:4090) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  |     at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:75) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  |     at reactor.core.publisher.Operators.complete(Operators.java:132) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  |     at reactor.core.publisher.MonoEmpty.subscribe(MonoEmpty.java:45) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  |     at reactor.core.publisher.Mono.subscribe(Mono.java:4105) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  |     at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:418) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  |     at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  |     at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.runAsync(FluxPublishOn.java:398) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  |     at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.run(FluxPublishOn.java:484) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  |     at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  |     at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) [reactor-core-3.3.2.RELEASE.jar!/:3.3.2.RELEASE]
kafka-ui  |     at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
kafka-ui  |     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
kafka-ui  |     at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
kafka-ui  |     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
kafka-ui  |     at java.lang.Thread.run(Unknown Source) [?:?]
kafka-ui  | Caused by: java.lang.IllegalArgumentException: Login module control flag not specified in JAAS config
kafka-ui  |     at org.apache.kafka.common.security.JaasConfig.parseAppConfigurationEntry(JaasConfig.java:110) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  |     at org.apache.kafka.common.security.JaasConfig.<init>(JaasConfig.java:63) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  |     at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:88) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  |     at org.apache.kafka.common.security.JaasContext.loadClientContext(JaasContext.java:82) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  |     at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:167) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  |     at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  |     at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  |     at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:513) ~[kafka-clients-2.8.0.jar!/:?]
kafka-ui  |     ... 22 more
Haarolean commented 3 years ago

Hey, no problem at all, happens to the best of us :)

Considering this line kafka-ui | Caused by: java.lang.IllegalArgumentException: Login module control flag not specified in JAAS config I raise a question: doesn't your username or password contain a semicolon by any chance? The parser might fail because it will try to consider it a second configuration entry.

harlev commented 3 years ago

No semicolon in the username or password. There are '/' and '+' characters though if that is an issue

Haarolean commented 3 years ago

Is that possible to try it out without such symbols? Despite having these values quoted they might get treated as special symbols.

harlev commented 3 years ago

I have no control of the username/password as I'm in an enterprise environment, also controlled by confluent as a service provider. Is there a way to escape these characters?

Haarolean commented 3 years ago

Okay, I get it. I'll try to reproduce the issue first. How do you run it there? K8s? Docker-compose?

harlev commented 3 years ago

I'm running in docker-compose locally against the Kafka service managed by Confluent on Azure

Haarolean commented 3 years ago

Weird, it works for me, at least with special symbols in password (I've tried usual kafka tho). Btw, why are your names/values separated by =? It's a colon which is used in docker compose, have you just replaced them or?

The question is, if you replace the username and password with something without special symbols, does it say authentication failed on start or is it the same "control flag" message?

UPD: I've got confluent kafka set up on azure, I'll get a chance to test it out soon. Stay tuned.

harlev commented 3 years ago

Thanks for the tips. After finding few places where " was replaced with “ and similar, and also converting from = to colons format, I got to the point I did get "Authentication Failed" when the password was just alpha characters. Now I'm v0.2.1 BTW

Then when I put the correct user/password I get org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.. For the record I manage to use the same credential to read and write to Kafka with other tools.

kafka-ui  | 19:33:24.268 [main] INFO  org.springframework.core.KotlinDetector - Kotlin reflection implementation not found at runtime, related features won't be available.
kafka-ui  |
kafka-ui  |   _  __      __ _               _   _ ___
kafka-ui  |  | |/ /__ _ / _| | ____ _      | | | |_ _|
kafka-ui  |  | ' // _` | |_| |/ / _` |_____| | | || |
kafka-ui  |  | . \ (_| |  _|   < (_| |_____| |_| || |
kafka-ui  |  |_|\_\__,_|_| |_|\_\__,_|      \___/|___|
kafka-ui  |
kafka-ui  |
kafka-ui  | 19:33:24.853 [main] INFO  com.provectus.kafka.ui.KafkaUiApplication - Starting KafkaUiApplication on f3bff025220f with PID 1 (/kafka-ui-api.jar started by root in /)
kafka-ui  | 19:33:24.853 [main] DEBUG com.provectus.kafka.ui.KafkaUiApplication - Running with Spring Boot v2.2.4.RELEASE, Spring v5.2.3.RELEASE
kafka-ui  | 19:33:24.854 [main] INFO  com.provectus.kafka.ui.KafkaUiApplication - No active profile set, falling back to default profiles: default
kafka-ui  | 19:33:25.121 [background-preinit] WARN  org.springframework.http.converter.json.Jackson2ObjectMapperBuilder - For Jackson Kotlin classes support please add "com.fasterxml.jackson.module:jackson-module-kotlin" to the classpath
kafka-ui  | SLF4J: Class path contains multiple SLF4J bindings.
kafka-ui  | SLF4J: Found binding in [jar:file:/kafka-ui-api.jar!/BOOT-INF/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
kafka-ui  | SLF4J: Found binding in [jar:file:/kafka-ui-api.jar!/BOOT-INF/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
kafka-ui  | SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
kafka-ui  | SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
kafka-ui  | log4j:WARN No appenders could be found for logger (reactor.util.Loggers$LoggerFactory).
kafka-ui  | log4j:WARN Please initialize the log4j system properly.
kafka-ui  | log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
kafka-ui  | 19:33:26.300 [main] INFO  org.springframework.boot.autoconfigure.security.reactive.ReactiveUserDetailsServiceAutoConfiguration -
kafka-ui  |
kafka-ui  | Using generated security password: ********
kafka-ui  |
kafka-ui  | 19:33:26.357 [main] WARN  org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JodaDateTimeJacksonConfiguration - Auto-configuration of Jackson's Joda-Time integration is deprecated in favor of using java.time (JSR-310).
kafka-ui  | 19:33:26.516 [main] INFO  com.provectus.kafka.ui.serde.DeserializationService - Using SchemaRegistryAwareRecordSerDe for cluster 'azureDev'
kafka-ui  | 19:33:27.116 [main] INFO  org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler - Initializing ExecutorService 'taskScheduler'
kafka-ui  | 19:33:27.206 [parallel-1] DEBUG com.provectus.kafka.ui.service.MetricsUpdateService - Start getting metrics for kafkaCluster: azureDev
kafka-ui  | 19:33:27.372 [main] INFO  org.springframework.boot.web.embedded.netty.NettyWebServer - Netty started on port(s): 8080
kafka-ui  | 19:33:27.377 [main] INFO  com.provectus.kafka.ui.KafkaUiApplication - Started KafkaUiApplication in 3.053 seconds (JVM running for 4.237)
kafka-ui  | 19:33:29.510 [kafka-admin-client-thread | adminclient-1] ERROR com.provectus.kafka.ui.service.KafkaService - Failed to collect cluster azureDev info
kafka-ui  | org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.
Haarolean commented 3 years ago

Eh, this looks unfortunate. I’ll try to reproduce with my azure instance tomorrow.

On 7 Oct 2021, at 23:00, Ron Harlev @.***> wrote:

 Thanks for the tips. After finding few places where " was replaced with “ and similar, and also converting from = to colons format, I got to the point I did get "Authentication Failed" when the password was just alpha characters.

Then when I put the correct user/password I get org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.. For the record I manage to use the same credential to read and write to Kafka with other tools.

kafka-ui | 19:33:24.268 [main] INFO org.springframework.core.KotlinDetector - Kotlin reflection implementation not found at runtime, related features won't be available. kafka-ui | kafka-ui | _ kafka-ui | | |/ / / | | __ | | | | | kafka-ui | | ' // | |_| |/ / _ |___| | | || | kafka-ui | | . \ (| | | < (| |____| || || | kafka-ui | ||__,|| ||__,| ___/|___| kafka-ui | kafka-ui | kafka-ui | 19:33:24.853 [main] INFO com.provectus.kafka.ui.KafkaUiApplication - Starting KafkaUiApplication on f3bff025220f with PID 1 (/kafka-ui-api.jar started by root in /) kafka-ui | 19:33:24.853 [main] DEBUG com.provectus.kafka.ui.KafkaUiApplication - Running with Spring Boot v2.2.4.RELEASE, Spring v5.2.3.RELEASE kafka-ui | 19:33:24.854 [main] INFO com.provectus.kafka.ui.KafkaUiApplication - No active profile set, falling back to default profiles: default kafka-ui | 19:33:25.121 [background-preinit] WARN org.springframework.http.converter.json.Jackson2ObjectMapperBuilder - For Jackson Kotlin classes support please add "com.fasterxml.jackson.module:jackson-module-kotlin" to the classpath kafka-ui | SLF4J: Class path contains multiple SLF4J bindings. kafka-ui | SLF4J: Found binding in [jar:file:/kafka-ui-api.jar!/BOOT-INF/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] kafka-ui | SLF4J: Found binding in [jar:file:/kafka-ui-api.jar!/BOOT-INF/lib/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] kafka-ui | SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. kafka-ui | SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] kafka-ui | log4j:WARN No appenders could be found for logger (reactor.util.Loggers$LoggerFactory). kafka-ui | log4j:WARN Please initialize the log4j system properly. kafka-ui | log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. kafka-ui | 19:33:26.300 [main] INFO org.springframework.boot.autoconfigure.security.reactive.ReactiveUserDetailsServiceAutoConfiguration - kafka-ui | kafka-ui | Using generated security password: 22516340-c209-4e09-8a5b-0bc5b8565133 kafka-ui | kafka-ui | 19:33:26.357 [main] WARN org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JodaDateTimeJacksonConfiguration - Auto-configuration of Jackson's Joda-Time integration is deprecated in favor of using java.time (JSR-310). kafka-ui | 19:33:26.516 [main] INFO com.provectus.kafka.ui.serde.DeserializationService - Using SchemaRegistryAwareRecordSerDe for cluster 'azureDev' kafka-ui | 19:33:27.116 [main] INFO org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler - Initializing ExecutorService 'taskScheduler' kafka-ui | 19:33:27.206 [parallel-1] DEBUG com.provectus.kafka.ui.service.MetricsUpdateService - Start getting metrics for kafkaCluster: azureDev kafka-ui | 19:33:27.372 [main] INFO org.springframework.boot.web.embedded.netty.NettyWebServer - Netty started on port(s): 8080 kafka-ui | 19:33:27.377 [main] INFO com.provectus.kafka.ui.KafkaUiApplication - Started KafkaUiApplication in 3.053 seconds (JVM running for 4.237) kafka-ui | 19:33:29.510 [kafka-admin-client-thread | adminclient-1] ERROR com.provectus.kafka.ui.service.KafkaService - Failed to collect cluster azureDev info kafka-ui | org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

Haarolean commented 3 years ago

Hi, sorry for the delay. I got a confluent kafka with azure and my setup works fine for me. That's how my docker-compose looks:

    environment:
      KAFKA_CLUSTERS_0_NAME: azureDev
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: pkc-lz6r3.northeurope.azure.confluent.cloud:9092
      KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SASL_SSL
      KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM: PLAIN
      KAFKA_CLUSTERS_0_PROPERTIES_CLIENT_DNS_LOOKUP: use_all_dns_ips
      KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG: "org.apache.kafka.common.security.plain.PlainLoginModule required username='xxx' password='yyy';"
      KAFKA_CLUSTERS_0_DISABLELOGDIRSCOLLECTION: 'true'

Please notice that quotes around username and passord are single and the ones around the whole string are double quotes. You could try to copy paste this and use your credentials. Let me know how it goes!

harlev commented 3 years ago

Thanks for testing it!!!

I copied your configuration as is, only replacing the bootstrap server name, username and password. I still get org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.

I changed the username to a wrong one, just to see the difference. I get, as expected org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed

May be something specific to our cluster. Willing to give up at this point :(

Just as a reference point, I'm using https://www.kafkatool.com/ with the exact same settings successfully.

Haarolean commented 3 years ago

Don’t give up yet. Since it works for me, I’ll provide you a test build with extra debugging to check it out, deal?

On 11 Oct 2021, at 21:32, Ron Harlev @.***> wrote:

 Thanks for testing it!!!

I copied your configuration as is, only replacing the bootstrap server name, username and password. I still get org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.

I changed the username to a wrong one, just to see the difference. I get, as expected org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed

May be something specific to our cluster. Willing to give up at this point :(

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

Haarolean commented 3 years ago

Ah, it seems like the issue is about permissions of some kind, not the authentication anymore.

Have you changed any ACLs/permissions on your cluster? Is that possible to check out the logs? Unfortunately kafka itself doesn't show the code in this exception, so that's not much of a help.

harlev commented 3 years ago

Since https://www.kafkatool.com/ is working with the same configuration and credentials, there is maybe some action you are using that is specifically blocked. Maybe there is a way to disable it.

I found few examples of people having this error: https://stackoverflow.com/questions/69070353/micronaut-kafka-health-check-fails-with-cluster-authorization-failed https://forum.confluent.io/t/admin-clusterauthorizationexception/1738 https://stackoverflow.com/questions/67148019/kafka-producer-property-enable-idempotence-true-is-causing-error

Haarolean commented 3 years ago

Yeah I thought the same, but I can't identify what does it do. As we agreed, it works on a default confluent-azure cluster.

Could you check which default permissions your setup can lack?

Does your user have describe clusters permission? We do that one for sure.

harlev commented 3 years ago

I don't have access to see or control what permissions I have :(

Haarolean commented 3 years ago

Is that possible to contact azure/confluent support then? We surely use describe clusters permission, maybe there's a chance you don't have it, idk.

harlev commented 3 years ago

Sorry, I currently don't have the bandwidth to deal with this. Will have to live without your great tool for now. I really appreciate the level of support your are providing!!!

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

dmcarba commented 2 years ago

Hello,

I've found this post because I was getting the same error: org.apache.kafka.common.errors.ClusterAuthorizationException. For me the problem was the ACLs of the user, you need to enable both describe and describe config for the Cluster in order to remove this error.

That's a problem because there are different tenants accessing the cluster with segregated ACLs permission on topics and group ids based on different prefix strings per each tenant. Allowing this cluster visibility is a potential risk, since each tenant can then see the other topic names and ACLs.

I wonder if there is the possibility of making the access to the extra cluster information optional so we can still use kafka-ui without allowing these permissions.

Thanks

relief-melone commented 2 years ago

Hello,

I've found this post because I was getting the same error: org.apache.kafka.common.errors.ClusterAuthorizationException. For me the problem was the ACLs of the user, you need to enable both describe and describe config for the Cluster in order to remove this error.

That's a problem because there are different tenants accessing the cluster with segregated ACLs permission on topics and group ids based on different prefix strings per each tenant. Allowing this cluster visibility is a potential risk, since each tenant can then see the other topic names and ACLs.

I wonder if there is the possibility of making the access to the extra cluster information optional so we can still use kafka-ui without allowing these permissions.

Thanks

I'd definately support this. There might be environments where this level of permissions is not given but it should not be necessary to view your topics and post message. So a "Non-Admin" mode would also help us very much

Haarolean commented 2 years ago

Hello, I've found this post because I was getting the same error: org.apache.kafka.common.errors.ClusterAuthorizationException. For me the problem was the ACLs of the user, you need to enable both describe and describe config for the Cluster in order to remove this error. That's a problem because there are different tenants accessing the cluster with segregated ACLs permission on topics and group ids based on different prefix strings per each tenant. Allowing this cluster visibility is a potential risk, since each tenant can then see the other topic names and ACLs. I wonder if there is the possibility of making the access to the extra cluster information optional so we can still use kafka-ui without allowing these permissions. Thanks

I'd definately support this. There might be environments where this level of permissions is not given but it should not be necessary to view your topics and post message. So a "Non-Admin" mode would also help us very much

We'll implement fine-grained access within #753

nuria commented 2 years ago

Leaving this here for posterity. If you are running against confluent cloud and you have specified correctly the jass config and still continue getting these errors look to to see if you are passing confluent.license in the connector, absence of a license returns a number of bogus errors like "Login module control flag not specified in JAAS config".

https://docs.confluent.io/platform/current/connect/license.html

Good resource for what properties are needed here: https://gist.github.com/rmoff/49526672990f1b4f7935b62609f6f567

Haarolean commented 2 years ago

If you are running against confluent cloud and you have specified correctly the jass config and still continue getting these errors look to to see if you are passing confluent.license in the connector, absence of a license returns a number of bogus errors like "Login module control flag not specified in JAAS config".

https://docs.confluent.io/platform/current/connect/license.html

Good resource for what properties are needed here: https://gist.github.com/rmoff/49526672990f1b4f7935b62609f6f567

thanks, added this into FAQ

dianadujing commented 1 year ago

Hi, sorry for the delay. I got a confluent kafka with azure and my setup works fine for me. That's how my docker-compose looks:

    environment:
      KAFKA_CLUSTERS_0_NAME: azureDev
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: pkc-lz6r3.northeurope.azure.confluent.cloud:9092
      KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SASL_SSL
      KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM: PLAIN
      KAFKA_CLUSTERS_0_PROPERTIES_CLIENT_DNS_LOOKUP: use_all_dns_ips
      KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG: "org.apache.kafka.common.security.plain.PlainLoginModule required username='xxx' password='yyy';"
      KAFKA_CLUSTERS_0_DISABLELOGDIRSCOLLECTION: 'true'

Please notice that quotes around username and passord are single and the ones around the whole string are double quotes. You could try to copy paste this and use your credentials. Let me know how it goes!

Thank you for sharing your solution. I resolved the same problem by following your instructions. Thanks!