search

aws / aws-msk-iam-auth

Enables developers to use AWS Identity and Access Management (IAM) to connect to their Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters.
Apache License 2.0
145 stars 67 forks source link

Cannot connect to MSK Serverless as a consumer #66

Closed ghost closed 2 years ago

ghost commented 2 years ago

The producer can connect and appears to produce messages fine, but when I try to connect as a consumer I get access denied. Simulating the instance perms says that the role has full access to everything it needs.

I can run the commands provided with the kafka zip file successfully, and running the consumer script for the appropriate topic shows all the messages produced into msk.

orchestrator-springboot-1  | 2022-05-06 18:11:46.381  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 3.0.1
orchestrator-springboot-1  | 2022-05-06 18:11:46.389  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: 8e30984f43e64d8b
orchestrator-springboot-1  | 2022-05-06 18:11:46.392  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1651860706378
orchestrator-springboot-1  | 2022-05-06 18:11:46.458  INFO 1 --- [           main] o.a.k.clients.consumer.KafkaConsumer     : [Consumer clientId=consumer-orchestrators-1, groupId=orchestrators] Subscribed to topic(s): orchestrator
orchestrator-springboot-1  | 2022-05-06 18:11:46.557  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8081 (http) with context path ''
orchestrator-springboot-1  | 2022-05-06 18:11:46.617  INFO 1 --- [           main] co.tryterra.wsorchestrator.Application   : Started Application in 11.478 seconds (JVM running for 12.58)
orchestrator-springboot-1  | 2022-05-06 18:11:47.743  INFO 1 --- [ntainer#0-0-C-1] o.apache.kafka.common.network.Selector   : [Consumer clientId=consumer-orchestrators-1, groupId=orchestrators] Failed authentication with boot-8nowkifx.c3.kafka-serverless.eu-west-1.amazonaws.com/172.31.38.204 ([169adb52-0325-4e3c-89f0-eb2e9205ebee]: Access denied)
orchestrator-springboot-1  | 2022-05-06 18:11:47.749 ERROR 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-orchestrators-1, groupId=orchestrators] Connection to node -1 (boot-8nowkifx.c3.kafka-serverless.eu-west-1.amazonaws.com/172.31.38.204:9098) failed authentication due to: [169adb52-0325-4e3c-89f0-eb2e9205ebee]: Access denied
orchestrator-springboot-1  | 2022-05-06 18:11:47.749  WARN 1 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=consumer-orchestrators-1, groupId=orchestrators] Bootstrap broker boot-8nowkifx.c3.kafka-serverless.eu-west-1.amazonaws.com:9098 (id: -1 rack: null) disconnected
orchestrator-springboot-1  | 2022-05-06 18:11:47.754 ERROR 1 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : Authentication/Authorization Exception and no authExceptionRetryInterval set
orchestrator-springboot-1  | 
orchestrator-springboot-1  | org.apache.kafka.common.errors.SaslAuthenticationException: [169adb52-0325-4e3c-89f0-eb2e9205ebee]: Access denied
orchestrator-springboot-1  | 
orchestrator-springboot-1  | 2022-05-06 18:11:47.755 ERROR 1 --- [ntainer#0-0-C-1] o.s.k.l.KafkaMessageListenerContainer    : Fatal consumer exception; stopping container
orchestrator-springboot-1  | 2022-05-06 18:11:47.760  INFO 1 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-orchestrators-1, groupId=orchestrators] Resetting generation due to: consumer pro-actively leaving the group
orchestrator-springboot-1  | 2022-05-06 18:11:47.760  INFO 1 --- [ntainer#0-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-orchestrators-1, groupId=orchestrators] Request joining group due to: consumer pro-actively leaving the group

I am using version 1.1.3 of the library.

ghost commented 2 years ago

Resolved as an application configuration issue