search

aws / aws-msk-iam-auth

Enables developers to use AWS Identity and Access Management (IAM) to connect to their Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters.
Apache License 2.0
138 stars 65 forks source link

Can not access msk with latest version (2.1.0) #168

Open hoalt14 opened 2 months ago

hoalt14 commented 2 months ago

Hi friends,

My version of MSK cluster is 3.5.1, and my client is an EC2 instance with IAM role attach exactly policy

I can't work the SASL mechanism = AWS_MSK_IAM with version 2.1.0 (It can not load credentials)

Here is the error logs: ERROR org.apache.kafka.common.errors.SaslAuthenticationException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Failed to find AWS IAM Credentials [Caused by software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[AwsCredentialsProviderChain(credentialsProviders=[EnvironmentVariableCredentialsProvider(), SystemPropertyCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider()])]) ...

But i can work with version 1.1.1

Why? Maybe SDKv2 not working? Help me explain this issue.

Thank you!

jvdadda commented 2 months ago

Hi @hoalt14 , indeed, I miss something during migration, the fix is in PR: https://github.com/aws/aws-msk-iam-auth/pull/169

jogilsang commented 2 months ago

Hi @hoalt14 , indeed, I miss something during migration, the fix is in PR: #169

still not working 2.1.0 but, It works in 2.0.3

My environment is as follows

# input
kafka-topics.sh --bootstrap-server $MSK_BOOTSTRAP_ADDRESS --command-config /tmp/client_iam.properties --list

# output
[2024-05-01 16:27:36,380] WARN Exception loading credentials. Retry Attempts: 0 (software.amazon.msk.auth.iam.internals.MSKCredentialProvider)
software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[AwsCredentialsProviderChain(credentialsProviders=[EnvironmentVariableCredentialsProvider(), SystemPropertyCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider()])]) : [AwsCredentialsProviderChain(credentialsProviders=[EnvironmentVariableCredentialsProvider(), SystemPropertyCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider()]): Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[EnvironmentVariableCredentialsProvider(), SystemPropertyCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider()]) : [EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])): Profile file contained no credentials for profile 'default': ProfileFile(sections=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set.]]
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
        at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:130)
        at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.loadCredentialsWithRetry(MSKCredentialProvider.java:175)
        at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.resolveCredentials(MSKCredentialProvider.java:162)
        at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handleCallback(IAMClientCallbackHandler.java:99)
        at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handle(IAMClientCallbackHandler.java:77)
        at software.amazon.msk.auth.iam.internals.IAMSaslClient.generateClientMessage(IAMSaslClient.java:139)
        at software.amazon.msk.auth.iam.internals.IAMSaslClient.evaluateChallenge(IAMSaslClient.java:96)
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.lambda$createSaslToken$1(SaslClientAuthenticator.java:535)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:535)
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:434)
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendInitialToken(SaslClientAuthenticator.java:333)
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.authenticate(SaslClientAuthenticator.java:274)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:181)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:585)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1504)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1435)
        at java.base/java.lang.Thread.run(Thread.java:829)
jvdadda commented 2 months ago

@jogilsang The fix is not deployed inside the 2.1.0, so it is "normal" to have it, a maintainer should initiate a release

kloometropolis commented 1 month ago

Will this fix be released soon?

yermulnik commented 4 weeks ago

We're hitting the same. Can the release be expedited please?

hhkkxxx133 commented 4 weeks ago

Hello, we are targeting on the release by the end of June. Thank you for your continued interest!

guanlinz commented 3 weeks ago

Hi there, Facing the same issue, here