How to support the refresh of IAM credentials without recreating the Kafka consumer

[j256]

Howdy. We are using the aws-msk-iam-auth jar with our Java Kafka components using the recommended awsRoleArn configurations for the IAMLoginModule in the sasl.jaas.config Kafka property which works initially but gets a "Topic authorization failed" error ("Not authorized to access topics") if the IAM credentials expire.

Is there a way to have the configurations automatically refresh? Are we missing a configuration option so that this happens automagically?

Thanks.

[sankalpbhatia]

Hi @j256,

If you are using Kafka clients version 2.2.0 and above, the client re-authentication flow should force re-authentication before the IAM credentials expire when using the awsRoleArn mechanism. If you are using a version < 2.2.0, the server will force close the connection after session expiry. So, the behavior you are seeing seems weird.

I would like to share a few pointers here.

1. Verify the awsRoleArn mechanism is indeed being used to generate the IAM auth payload. You can use the awsDebugCreds field in the sasl.jaas.config property to verify this.

For points 2 and 3, I am assuming you are using Kafka client versions 2.2.0 and above. Let us know if that's not the case.

2. When a connection is created, the server will notify the client about the session duration, and a session re-authentication duration. The DEBUG log in this case looks like this:

[2022-04-27 00:10:43,423] DEBUG [Producer clientId=producer-1] Finished authentication with session expiration in 744585 ms and session re-authentication on or after 634343 ms (org.apache.kafka.common.security.authenticator.SaslClientAuthenticator)

Would it be possible for you to verify that the Kafka client session expiration you see in client debug logs is same as the IAM role's session expiration time?

3. When the session expiration time approaches, do you see a re-authentication happening? Would be helpful to see what Identity is being used to generate the auth payload at that time.

Let us know if you any more help in debugging this.

[j256]

So sorry for the delay. Not sure why I didn't see your response until now.

Verify the awsRoleArn mechanism is indeed being used to generate the IAM auth payload. You can use the awsDebugCreds field in the sasl.jaas.config property to verify this.

Will do.

For points 2 and 3, I am assuming you are using Kafka client versions 2.2.0 and above. Let us know if that's not the case.

We are using 2.8.2.

Would it be possible for you to verify that the Kafka client session expiration you see in client debug logs is same as the IAM role's session expiration time?

I will look for these debug messages.

When the session expiration time approaches, do you see a re-authentication happening? Would be helpful to see what Identity is being used to generate the auth payload at that time.

Ok. I'll look for that as well.

Thanks much!

[j256]

We have been testing with 3.4.1 of kafka-clients. We are creating IAM keys with a 15 minute expiration and we see:

org.apache.kafka.common.network.Selector - [Consumer clientId=kafkautil, groupId=kafkautil]
    Successfully re-authenticated with b-1.str11521amsk.bdiu9d.c16.kafka.us-east-1.amazonaws.com/10.193.0.172

But at the end of the 15 minutes we still see:

aws_msk_iam_auth_shadow.com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
   The security token included in the request is expired (Service: AWSSecurityTokenService; Status Code: 403;
   Error Code: ExpiredToken; Request ID: 4df0f7fa-9e8a-4b53-a172-26b6763f8677; Proxy: null)

This may be user error around the creation of the iAM credentials. We think we are creating the credentials using the awsRoleArn mechanism, but the awsRoleArn attribute was not in our credentials profile so we are running our tests again. Is there anything more specific that we need to be doing there?

[sankalpbhatia]

We are creating IAM keys with a 15 minute expiration

How are these credentials used by the application? Do you intend to use these keys to assume the awsRoleArn?

From what you shared, it seems like the credentials used to refresh the role credentials (by calling STS client) are getting expired.

Can you also share your jaas.config properties ?

[dsinghal-nice]

Hi @sankalpbhatia,

I have a use case where I need to assume role in another account and consume MSK continuously, will the above approach work there as well for cross account access? Also will the credentials will get automatically refreshed without me doing anything? And last question is, Currently I am using 1.1.5 version of aws-msk-iam-auth library and it needs me to save the credentials in System properties and then the flow works fine, with the new version do I need to do STSAssumeRole or it's not even required with the awsRoleArn and the credentials will refresh automatically (Since I want to use other aws services from source account and when I set the credentials in System properties I fail to access the source resources)?