dynamodb/DescribeTimeToLive throttling error is not respected

[lukep-coxauto]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.2.6
on windows_amd64
+ provider registry.terraform.io/hashicorp/aws v4.24.0

Affected Resource(s)

• aws_dynamodb_table

Terraform Configuration Files

Our AWS account has 36 DynamoDB tables that were deployed with a previous version of the AWS provider. When we run terraform plan to refresh the state, we receive throttling errors when it is describing the TTL values for these tables.

Debug Output

Turning on Debug/Trace doesn't make any difference, the plan fails with a 400 bad request on the describe-time-to-live calls. Here are some example lines of output.

│ Error: reading Amazon DynamoDB Table (QaDevPortalBuildTracker): TTL: ThrottlingException: Rate exceeded
│       status code: 400, request id: HPFSK782M4RPV5NMQ73FO0PJA3VV4KQNSO5AEMVJF66Q9ASUAAJG
│
│   with aws_dynamodb_table.QaDevPortalBuildTracker-dynamodb,
│   on cdk.tf.json line 2156, in resource.aws_dynamodb_table.QaDevPortalBuildTracker-dynamodb:
│ 2156:       },
│
╵
╷
│ Error: reading Amazon DynamoDB Table (QaDevPortalDeployLog): TTL: ThrottlingException: Rate exceeded
│       status code: 400, request id: D8EFATIKCS145PCFQB03AJKCFFVV4KQNSO5AEMVJF66Q9ASUAAJG
│
│   with aws_dynamodb_table.QaDevPortalDeployLog-dynamodb,
│   on cdk.tf.json line 2210, in resource.aws_dynamodb_table.QaDevPortalDeployLog-dynamodb:
│ 2210:       },
│

Panic Output

Expected Behavior

terraform plan should properly wait for throttling related errors.

Actual Behavior

terraform plan fails unless you run it with a lower parallelism to avoid the rate limit issue. terraform plan -parallelism=4 also failed. terraform plan -parallelism=1 is successful, indicating the underlying cause is in fact rate limit-related.

Steps to Reproduce

1. run terraform plan against an account with many existing DynamoDB tables (in our case, 36). getting the TTL property fails due to a rate limit error.

Important Factoids

I think part of the issue is that the DescribeTimeToLive is not returning a 429, but instead a 400, so the exponential backoff can't be run to compensate for the rate limit being exceeded. Adjusting the retry policy of the provider appeared to have no effect (raised retries from 2 to 10 and saw the same issue).

References

[github-actions[bot]]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!