search

aws / aws-msk-iam-auth

Enables developers to use AWS Identity and Access Management (IAM) to connect to their Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters.
Apache License 2.0
145 stars 67 forks source link

kafka-consumer-topics.sh --list returns empty list #192

Open af6140 opened 1 week ago

af6140 commented 1 week ago

Client is kafa 3.3.1 with latest version of this library running as ECS task with IAM policy for task setup. AWS MSK 2.7 with IAM enabled. The command returns empty topic list

[2024-11-14 00:21:33,557] DEBUG Setting SASL/AWS_MSK_IAM.498931366 client state to RECEIVE_SERVER_RESPONSE (software.amazon.msk.auth.iam.internals.IAMSaslClient)
[2024-11-14 00:21:33,557] DEBUG State RECEIVE_SERVER_RESPONSE at end of evaluating challenge (software.amazon.msk.auth.iam.internals.IAMSaslClient)
[2024-11-14 00:21:33,558] DEBUG [AdminClient clientId=adminclient-1] Set SASL client state to INTERMEDIATE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2024-11-14 00:21:33,594] DEBUG State RECEIVE_SERVER_RESPONSE at start of evaluating challenge (software.amazon.msk.auth.iam.internals.IAMSaslClient)
[2024-11-14 00:21:33,597] DEBUG Response from server: AuthenticationResponse(version=2020_10_22, requestId=xxxxxx) (software.amazon.msk.auth.iam.internals.IAMSaslClient)
[2024-11-14 00:21:33,597] DEBUG Setting SASL/AWS_MSK_IAM.498931366 client state to COMPLETE (software.amazon.msk.auth.iam.internals.IAMSaslClient)
[2024-11-14 00:21:33,597] DEBUG State COMPLETE at end of evaluating challenge (software.amazon.msk.auth.iam.internals.IAMSaslClient)
[2024-11-14 00:21:33,597] DEBUG [AdminClient clientId=adminclient-1] Set SASL client state to COMPLETE (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2024-11-14 00:21:33,597] DEBUG [AdminClient clientId=adminclient-1] Finished authentication with session expiration in 20921407 ms and session re-authentication on or after 19064973 ms (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)
[2024-11-14 00:21:33,597] DEBUG [AdminClient clientId=adminclient-1] Successfully authenticated with b-2.xxxxxxkafka.us-east-1.amazonaws.com/10.234.236.25 (org.apache.kafka.common.network.Selector)
[2024-11-14 00:21:33,597] DEBUG [AdminClient clientId=adminclient-1] Initiating API versions fetch from node 2. (org.apache.kafka.clients.NetworkClient)
[2024-11-14 00:21:33,597] DEBUG [AdminClient clientId=adminclient-1] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=2) and timeout 3600000 to node 2: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='3.3.1') (org.apache.kafka.clients.NetworkClient)
[2024-11-14 00:21:33,602] DEBUG [AdminClient clientId=adminclient-1] Received API_VERSIONS response from node 2 for request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=2): ApiVersionsResponseData(errorCode=0, apiKeys=[ApiVersion(apiKey=0, minVersion=0, maxVersion=8), ApiVersion(apiKey=1, minVersion=0, maxVersion=12), ApiVersion(apiKey=2, minVersion=0, maxVersion=5), ApiVersion(apiKey=3, minVersion=0, maxVersion=9), ApiVersion(apiKey=4, minVersion=0, maxVersion=4), ApiVersion(apiKey=5, minVersion=0, maxVersion=3), ApiVersion(apiKey=6, minVersion=0, maxVersion=6), ApiVersion(apiKey=7, minVersion=0, maxVersion=3), ApiVersion(apiKey=8, minVersion=0, maxVersion=8), ApiVersion(apiKey=9, minVersion=0, maxVersion=7), ApiVersion(apiKey=10, minVersion=0, maxVersion=3), ApiVersion(apiKey=11, minVersion=0, maxVersion=7), ApiVersion(apiKey=12, minVersion=0, maxVersion=4), ApiVersion(apiKey=13, minVersion=0, maxVersion=4), ApiVersion(apiKey=14, minVersion=0, maxVersion=5), ApiVersion(apiKey=15, minVersion=0, maxVersion=5), ApiVersion(apiKey=16, minVersion=0, maxVersion=4), ApiVersion(apiKey=17, minVersion=0, maxVersion=1), ApiVersion(apiKey=18, minVersion=0, maxVersion=3), ApiVersion(apiKey=19, minVersion=0, maxVersion=6), ApiVersion(apiKey=20, minVersion=0, maxVersion=5), ApiVersion(apiKey=21, minVersion=0, maxVersion=2), ApiVersion(apiKey=22, minVersion=0, maxVersion=4), ApiVersion(apiKey=23, minVersion=0, maxVersion=3), ApiVersion(apiKey=24, minVersion=0, maxVersion=2), ApiVersion(apiKey=25, minVersion=0, maxVersion=2), ApiVersion(apiKey=26, minVersion=0, maxVersion=2), ApiVersion(apiKey=27, minVersion=0, maxVersion=0), ApiVersion(apiKey=28, minVersion=0, maxVersion=3), ApiVersion(apiKey=29, minVersion=0, maxVersion=2), ApiVersion(apiKey=30, minVersion=0, maxVersion=2), ApiVersion(apiKey=31, minVersion=0, maxVersion=2), ApiVersion(apiKey=32, minVersion=0, maxVersion=3), ApiVersion(apiKey=33, minVersion=0, maxVersion=1), ApiVersion(apiKey=34, minVersion=0, maxVersion=1), ApiVersion(apiKey=35, minVersion=0, maxVersion=2), ApiVersion(apiKey=36, minVersion=0, maxVersion=2), ApiVersion(apiKey=37, minVersion=0, maxVersion=3), ApiVersion(apiKey=38, minVersion=0, maxVersion=2), ApiVersion(apiKey=39, minVersion=0, maxVersion=2), ApiVersion(apiKey=40, minVersion=0, maxVersion=2), ApiVersion(apiKey=41, minVersion=0, maxVersion=2), ApiVersion(apiKey=42, minVersion=0, maxVersion=2), ApiVersion(apiKey=43, minVersion=0, maxVersion=2), ApiVersion(apiKey=44, minVersion=0, maxVersion=1), ApiVersion(apiKey=45, minVersion=0, maxVersion=0), ApiVersion(apiKey=46, minVersion=0, maxVersion=0), ApiVersion(apiKey=47, minVersion=0, maxVersion=0), ApiVersion(apiKey=48, minVersion=0, maxVersion=0), ApiVersion(apiKey=49, minVersion=0, maxVersion=0), ApiVersion(apiKey=50, minVersion=0, maxVersion=0), ApiVersion(apiKey=51, minVersion=0, maxVersion=0), ApiVersion(apiKey=56, minVersion=0, maxVersion=0), ApiVersion(apiKey=57, minVersion=0, maxVersion=0)], throttleTimeMs=0, supportedFeatures=[], finalizedFeaturesEpoch=0, finalizedFeatures=[]) (org.apache.kafka.clients.NetworkClient)
[2024-11-14 00:21:33,603] DEBUG [AdminClien clientId=adminclient-1] Node 2 has finalized features epoch: 0, finalized features: [], supported features: [], API versions: (Produce(0): 0 to 8 [usable: 8], Fetch(1): 0 to 12 [usable: 12], ListOffsets(2): 0 to 5 [usable: 5], Metadata(3): 0 to 9 [usable: 9], LeaderAndIsr(4): 0 to 4 [usable: 4], StopReplica(5): 0 to 3 [usable: 3], UpdateMetadata(6): 0 to 6 [usable: 6], ControlledShutdown(7): 0 to 3 [usable: 3], OffsetCommit(8): 0 to 8 [usable: 8], OffsetFetch(9): 0 to 7 [usable: 7], FindCoordinator(10): 0 to 3 [usable: 3], JoinGroup(11): 0 to 7 [usable: 7], Heartbeat(12): 0 to 4 [usable: 4], LeaveGroup(13): 0 to 4 [usable: 4], SyncGroup(14): 0 to 5 [usable: 5], DescribeGroups(15): 0 to 5 [usable: 5], ListGroups(16): 0 to 4 [usable: 4], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 3 [usable: 3], CreateTopics(19): 0 to 6 [usable: 6], DeleteTopics(20): 0 to 5 [usable: 5], DeleteRecords(21): 0 to 2 [usable: 2], InitProducerId(22): 0 to 4 [usable: 4], OffsetForLeaderEpoch(23): 0 to 3 [usable: 3], AddPartitionsToTxn(24): 0 to 2 [usable: 2], AddOffsetsToTxn(25): 0 to 2 [usable: 2], EndTxn(26): 0 to 2 [usable: 2], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 to 3 [usable: 3], DescribeAcls(29): 0 to 2 [usable: 2], CreateAcls(30): 0 to 2 [usable: 2], DeleteAcls(31): 0 to 2 [usable: 2], DescribeConfigs(32): 0 to 3 [usable: 3], AlterConfigs(33): 0 to 1 [usable: 1], AlterReplicaLogDirs(34): 0 to 1 [usable: 1], DescribeLogDirs(35): 0 to 2 [usable: 2], SaslAuthenticate(36): 0 to 2 [usable: 2], CreatePartitions(37): 0 to 3 [usable: 3], CreateDelegationToken(38): 0 to 2 [usable: 2], RenewDelegationToken(39): 0 to 2 [usable: 2], ExpireDelegationToken(40): 0 to 2 [usable: 2], DescribeDelegationToken(41): 0 to 2 [usable: 2], DeleteGroups(42): 0 to 2 [usable: 2], ElectLeaders(43): 0 to 2 [usable: 2], IncrementalAlterConfigs(44): 0 to 1 [usable: 1], AlterPartitionReassignments(45): 0 [usable: 0], ListPartitionReassignments(46): 0 [usable: 0], OffsetDelete(47): 0 [usable: 0], DescribeClientQuotas(48): 0 [usable: 0], AlterClientQuotas(49): 0 [usable: 0], DescribeUserScramCredentials(50): 0 [usable: 0], AlterUserScramCredentials(51): 0 [usable: 0], AlterPartition(56): 0 [usable: 0], UpdateFeatures(57): 0 [usable: 0], DescribeCluster(60): UNSUPPORTED, DescribeProducers(61): UNSUPPORTED, DescribeTransactions(65): UNSUPPORTED, ListTransactions(66): UNSUPPORTED, AllocateProducerIds(67): UNSUPPORTED). (org.apache.kafka.clients.NetworkClient)
[2024-11-14 00:21:33,604] DEBUG [AdminClient clientId=adminclient-1] Sending MetadataRequestData(topics=null, allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to b-2.xxxxxx.kafka.us-east-1.amazonaws.com:9098 (id: 2 rack: use1-az6). correlationId=3, timeoutMs=29779 (org.apache.kafka.clients.admin.KafkaAdminClient)
[2024-11-14 00:21:33,604] DEBUG [AdminClient clientId=adminclient-1] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=9, clientId=adminclient-1, correlationId=3) and timeout 29779 to node 2: MetadataRequestData(topics=null, allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) (org.apache.kafka.clients.NetworkClient)
[2024-11-14 00:21:33,971] DEBUG [AdminClient clientId=adminclient-1] Received METADATA response from node 2 for request with header RequestHeader(apiKey=METADATA, apiVersion=9, clientId=adminclient-1, correlationId=3): MetadataResponseData(throttleTimeMs=0, brokers=[MetadataResponseBroker(nodeId=2, host='b-2.xxxxxkafka.us-east-1.amazonaws.com', port=9098, rack='use1-az6'), MetadataResponseBroker(nodeId=3, host='b-3.xxxxxx.kafka.us-east-1.amazonaws.com', port=9098, rack='use1-az2'), MetadataResponseBroker(nodeId=1, host='b-1.xxxxxxkafka.us-east-1.amazonaws.com', port=9098, rack='use1-az4')], clusterId='12345', controllerId=2, topics=[], clusterAuthorizedOperations=-2147483648) (org.apache.kafka.clients.NetworkClient)

[2024-11-14 00:21:33,980] DEBUG [AdminClient clientId=adminclient-1] Initiating close operation. (org.apache.kafka.clients.admin.KafkaAdminClient)
[2024-11-14 00:21:33,981] DEBUG [AdminClient clientId=adminclient-1] Waiting for the I/O thread to exit. Hard shutdown in 31535999999 ms. (org.apache.kafka.clients.admin.KafkaAdminClient)
[2024-11-14 00:21:33,981] INFO App info kafka.admin.client for adminclient-1 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2024-11-14 00:21:33,990] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
[2024-11-14 00:21:33,993] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
[2024-11-14 00:21:33,993] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
[2024-11-14 00:21:33,993] DEBUG [AdminClient clientId=adminclient-1] Exiting AdminClientRunnable thread. (org.apache.kafka.clients.admin.KafkaAdminClient)
[2024-11-14 00:21:33,994] DEBUG [AdminClient clientId=adminclient-1] Kafka admin client closed. (org.apache.kafka.clients.admin.KafkaAdminClient)