The security token included in the request is expired - JAVA SDK

[vickeyrihal1]

Describe the bug Observing one of our container fail to fetch Message from SQS with error 'The security token included in the request is expired'. This had happened with multiple times for brief period with multiple services (SQS+ Dynamo DB). Issue got resolved exactly after 15 minutes, without intervening anything.

AWS java SDK - aws-java-sdk-core-1.12.201 Using InstanceProfileCredentialsProvider = new InstanceProfileCredentialsProvider(true)); Cluster: Kubernetes v1.21.11 IAM component: kube2iam (https://github.com/jtblin/kube2iam) Kube2iam version: 0.10.11

ERROR log: com.amazonaws.services.sqs.model.AmazonSQSException: The security token included in the request is expired (Service: AmazonSQS; Status Code: 403; Error Code: ExpiredToken; Request ID: 25b9b369-9954-5d2c-8c21-e84006d4ec55; Proxy: null).

Expected Behavior Issue should not be occurring intermittently. SDK should always be able to fetch token before it get expired.

Current Behavior On of container fail to fetch Message from SQS with error 'The security token included in the request is expired'. This had happened with multiple times for brief period with multiple se

Reproduction Steps Intermittently. Not always producible.

AWS Java SDK version used aws-java-sdk-core-1.12.201

JDK version used java version "11.0.7" 2020-04-14

Operating System and version Kubernetes v1.21.11

[vickeyrihal1]

Issue with SDK: https://github.com/aws/aws-sdk-java/issues/2802

Comment from maintainer: https://github.com/aws/aws-sdk-java/issues/2802#issuecomment-1210010470

[vickeyrihal1]

Please take a look at this comment as well. https://github.com/aws/aws-sdk-java/issues/2802#issuecomment-1220583337

[vickeyrihal1]

can we get some traction on this?

[shellshock1953]

Hey, we can see the same errors (Aws::SQS::Errors::ExpiredToken, or for S3, or DynamoDB) in Ruby:

aws-sdk-sqs (1.51.1)
aws-sdk-s3 (1.114.0)
aws-sdk-core (3.131.6)
kube2iam (0.10.9)

[shambhand]

We are facing the same issue; upon investigation, it founds that kube2iam is taking more than 1 second for some call For aws-java-sdk, the default connection timeout to IMDS service to fetch the credential is 1 second and it can be overridden with the environment variable AWS_METADATA_SERVICE_TIMEOUT from aws-java-sdk v>=1.12.39 2021-08-03.

After setting the AWS_METADATA_SERVICE_TIMEOUT env variable to 3 seconds; it's resolved for us.

https://github.com/aws/aws-sdk-java/discussions/2881

[jtblin]

kube2iam calls the IAM service to get the token, this is most likely why it takes more time to resolve from time to time. kube2iam is pretty lightweight so it's unlikely to be the cause of the timeouts.