AWS_MAX_ATTEMPTs and similar cannot be used to address AssumeRoleWithWebIdentity rate limits

[djmcgreal-cc]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Environment and Versions

• Terraform Executor: Local workstation / CI

  • If outside Terraform Cloud/Enterprise, which operating system and version: Linux/macOS

• Terraform CLI version: Terraform v1.2.4 - I've done most testing on earlier version but also seen on 1.3.7

• Terraform AWS Provider version: 4.30.0 - Likewise on 4.52.0

• Terraform Backend/Provider Configuration: S3

• AWS environment variables (if any):

AWS_MAX_ATTEMPTS=50
AWS_WEB_IDENTITY_TOKEN_FILE="/some/valid/jwt"
AWS_ROLE_ARN="arn:..."

Expected Behavior

AssumeRoleWithWebIdentity calls seem to be rate limited by AWS. When authenticating with AWS_WEB_IDENTITY_TOKEN_FILE, I'd hoped to work around this by setting AWS_MAX_ATTEMPTS.

Actual Behavior

Neither AWS_MAX_ATTEMPTS, nor max_retries field have any effect. I've debugged the code and found the issue.

aws-sdk-go-v2 (which this library uses) recognises the rate limiting by adding the InvalidIdentityToken error to the retryer, but this module nullifies the retryer (mode, max_retries, or both) for the purpose of resolving credentials.

Steps to Reproduce

1. Have an identity token that is configured to be able to assume a role with AWS
2. Set AWS_WEB_IDENTITY_TOKEN_FILE and run multiple quick authentications via e.g. terraform plan with a simple configuration

References

See inline

[jBouyoud]

Hi,

I'm facing the same issue and this is a handicap. Seems related to another issue on provider side. @gdavison, I'll be glad to help fixing this issue, please let me know how to proceed.

I found a workaround, if you terraform is based on profile you can tweak a bit ~/.aws/config to do the retry in place of terraform provider. Your config file looks like :

[profile my-profile]
role_arn = arn:aws:iam::1234567890123:role/role_name
web_identity_token_file = /path/to/web_identity_token_file
role_session_name = blabla
region = eu-central-1

You can use credential_process to workaround the issue:

• Duplicate the profile in another one
• Use credential_process to load credential from the duplicated ones with a backoff retries

  
  [profile my-profile]
  credentials_process = with_backoff 10 aws configure export-credentials --format=process --profile=my-profile-raw
  region = eu-central-1

[profile my-profile-raw] role_arn = arn:aws:iam::1234567890123:role/role_name web_identity_token_file = /path/to/web_identity_token_file role_session_name = blabla region = eu-central-1

where `with_backoff` could be (in your path) :
```bash
#!/usr/bin/env bash

function with_backoff() {
  local max_attempts=$1
  local timeout=1
  local exitCode=0
  shift 1

  for attempt in $(seq 1 $max_attempts); do
    if [[ ${attempt} -gt 1 ]]; then
      echo "Trying [${attempt}/${max_attempts}] ($@)..." 1>&2
    fi
    "$@" && exitCode=$? || exitCode=$?

    if [[ ${exitCode} == 0 ]]; then
      return 0
    fi

    echo "Failure (${exitCode})! Retrying [${attempt}/${max_attempts}] in ${timeout}.." 1>&2
    sleep "${timeout}"
    timeout=$(( timeout * 2 ))
  done

  echo "You've failed (${exitCode}) me for the last time (${attempt}/${max_attempts})! ($@)" 1>&2
  return ${exitCode}
}

with_backoff $@

As all workaround this is sob, awaiting fixed by this issue 🤞