search

aws / aws-msk-iam-auth

Enables developers to use AWS Identity and Access Management (IAM) to connect to their Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters.
Apache License 2.0
141 stars 66 forks source link

Error connecting to MSK using SSO-based profile #31

Closed mrjoycee closed 3 years ago

mrjoycee commented 3 years ago

I have reproduced this error using Conduktor as well as the Kafka command line tools.

When configuring awsProfileName in the sasl.jaas.config to correspond to an AWS SSO-based profile, I am seeing the following error in the output: [2021-07-15 15:33:37,660] ERROR org.apache.kafka.common.errors.SaslAuthenticationException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Failed to find AWS IAM Credentials [Caused by aws_msk_iam_auth_shadow.com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [software.amazon.msk.auth.iam.internals.EnhancedProfileCredentialsProvider@3aa66d75: Unable to load an HTTP implementation from any provider in the chain. You must declare a dependency on an appropriate HTTP implementation or pass in an SdkHttpClient explicitly to the client builder.

I am able to successfully connect using environment variables generated via the "Command line or programmatic access" tool on the AWS SSO start page.

sayantacC commented 3 years ago

This indicates that the IAM library could not either find the specified profile or use the specified profile to generate credentials. Could you please provide the sasl.jaas.config you used as well as the config for the aws sso based profile from ~/.aws/config ?

sayantacC commented 3 years ago

I assume you are trying this with the uber jar. This may also be related to problems finding a SdkHttpClient on the classpath since the aws sdk libraries were relocated as part of creating the uber jar. Similar in effect to the issue described here, but different cause.

I will try to look into this.

mrjoycee commented 3 years ago

Yes I am using aws-msk-iam-auth-1.1.0-all.jar

mrjoycee commented 3 years ago

Profile config:

[profile platform-dev-kafka]
sso_start_url = <redacted> 
sso_region = us-east-1  
sso_account_id = <redacted> 
sso_role_name = Platform-Dev-Kafka-Admins 
region = us-east-1 
output = json

Kafka command config:

security.protocol=SASL_SSL 
sasl.mechanism=AWS_MSK_IAM 
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required awsProfileName="platform-dev-kafka";  
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
sayantacC commented 3 years ago

I have isolated the issue to a problem with relocating the aws sdk v2 (used to load the sso credentials) while generating the uber-jar. The relocation messes with different classes that are loaded by the aws sdk v2 dynamically. It will take some time to figure out what the right way of relocating the aws sdk v2 is such that the classloading does not break.

In the meanwhile, I plan to change the generation of the uber jar to include the aws sdk v2 but not relocate it. I do not expect kafka clients using the uber jar to have the aws sdk v2 classes in their path.

sayantacC commented 3 years ago

In my testing, SSO works with the uber-jar after this change.