search

aws / aws-msk-iam-auth

Enables developers to use AWS Identity and Access Management (IAM) to connect to their Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters.
Apache License 2.0
142 stars 65 forks source link

Classcast Exception when trying to connect to MSK using AWS_MSK_IAM from Datahub #50

Closed arunvasudevan closed 2 years ago

arunvasudevan commented 2 years ago

Team, I am trying to deploy Datahub project specifically GMS Service after adding the following jars to the container (aws-msk-iam-auth-1.1.1-all.jar, kafka-clients-2.3.0.jar, slf4j-api-1.7.31.jar). Datahub Version that i am working on is the latest one v0.8.21 I am trying to connect to AWS MSK bootstrap-servers using AWS_MSK_IAM SASL Mechanism.

But, I am getting a classcast exception in my service logs

Caused by: java.lang.ClassCastException: class software.amazon.msk.auth.iam.IAMClientCallbackHandler cannot be cast to class org.apache.kafka.common.security.auth.AuthenticateCallbackHandler (software.amazon.msk.auth.iam.IAMClientCallbackHandler is in unnamed module of loader java.net.URLClassLoader @73846619; org.apache.kafka.common.security.auth.AuthenticateCallbackHandler is in unnamed module of loader org.eclipse.jetty.webapp.WebAppClassLoader @667a738)
    at org.apache.kafka.common.network.SaslChannelBuilder.createClientCallbackHandler(SaslChannelBuilder.java:293)
    at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:136)

Following are the env variables added to the project to run 

SPRING_KAFKA_PROPERTIES_SECURITY_PROTOCOL=SASL_SSL
SPRING_KAFKA_PROPERTIES_SSL_TRUSTSTORE_LOCATION=/tmp/kafka.client.truststore.jks
SPRING_KAFKA_PROPERTIES_SASL_MECHANISM=AWS_MSK_IAM
SPRING_KAFKA_PROPERTIES_SASL_JAAS_CONFIG=software.amazon.msk.auth.iam.IAMLoginModule required awsDebugCreds=true;
SPRING_KAFKA_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS=software.amazon.msk.auth.iam.IAMClientCallbackHandler

Attached the logs.... gms_v0.8.21.log

Also attaching the custom Docker file and start script to add the above highlighted jars Scripts.zip

Any help on this would be great!

sayantacC commented 2 years ago

From the error message, it seems like there are two different class loaders involved in the application java.net.URLClassLoader @73846619 and org.eclipse.jetty.webapp.WebAppClassLoader @667a738.

That normally indicates that the Kafka client libraries and the aws-msk-iam-auth library are being loaded by two different class loaders. Sadly, I am not familiar enough with Datahub to know why two different class loaders are being used.

You could try running the gms service with the JVM argument -verbose:class to see if that provides any hint as to how the kafka clients and aws-msk-iam-auth libraries are being loaded. Please note that this will generate a lot of log lines.

Although I am not familiar with Datahub itself, in most cases where the aws-msk-iam-auth library is used as part of another framework the aws-msk-iam-auth library should be on the classpath and not some other path used to load plugins.

Hope this helps.

arunvasudevan commented 2 years ago

Thanks @sayantacC for your inputs.

Added aws-msk-iam-auth:1.1.1 as an implementation dependency to the metadata-service project.

Now, I am getting the below error....

Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [com.amazonaws.auth.AWSCredentialsProviderChain@4d9fa6ba: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), WebIdentityTokenCredentialsProvider: You must specify a value for roleArn and roleSessionName, software.amazon.msk.auth.iam.internals.EnhancedProfileCredentialsProvider@2f6a0f9c: Profile file contained no credentials for profile 'default': ProfileFile(profiles=[]), com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@50274fbd: The requested metadata is not found at http://<redact>/latest/meta-data/iam/security-credentials/]]
    at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:136)
    at software.amazon.msk.auth.iam.internals.MSKCredentialProvider.getCredentials(MSKCredentialProvider.java:86)
    at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handleCallback(IAMClientCallbackHandler.java:100)
    at software.amazon.msk.auth.iam.IAMClientCallbackHandler.handle(IAMClientCallbackHandler.java:77)
    at software.amazon.msk.auth.iam.internals.IAMSaslClient.generateClientMessage(IAMSaslClient.java:138)

From the stacktrace it looks like its able to find the MSKCredentialProvider but not able to find the credentials. Are there additional env variables that should be set? Not clear to me trying to find more documentation around this.

garystafford commented 2 years ago

@arunvasudevan if you are running DataHub on K8s (Amazon EKS), the IAM Roles for Service Accounts (IRSA) - the role associated with the pod will provide the proper IAM auth. to access Amazon MSK.

arunvasudevan commented 2 years ago

Thanks @garystafford and @sayantacC after adding the aws-msk-iam-auth library to the GMS Dependencies the issue was resolved.