RanVaknin
commented
7 months ago Hi @sfxandy,
This is indeed an interesting problem. By default the SDK does retry various rate limit exceptions:
var DefaultThrottleErrorCodes = map[string]struct{}{
"Throttling": {},
"ThrottlingException": {},
"ThrottledException": {},
"RequestThrottledException": {},
"TooManyRequestsException": {},
"ProvisionedThroughputExceededException": {},
"TransactionInProgressException": {},
"RequestLimitExceeded": {},
"BandwidthLimitExceeded": {},
"LimitExceededException": {},
"RequestThrottled": {},
"SlowDown": {},
"PriorRequestNotComplete": {},
"EC2ThrottledException": {},
}It seems like EKS is throwing some sort of throttling error based on the description, but neither the exception name (InvalidRequestException) nor the status code (400) indicate this. For a throttling error to be retried by the SDK, it has to have an error code of 429/5xx or a concrete exception that is one of those defined above.
Ideally, the EKS service should have returned a more accurate error, but changing returned data for services would be considered backwards incompatible and might break existing customers. In this case your only course of action would be to implement your own retryer and explicitly retry on "InvalidRequestException". You can refer to our docs for more information regarding info about the aws.Retryer.
Please let me know if you need anything else.
All the best, Ran~
sfxandy
lucix-aws
sergiomacedo
github-actions[bot]
Describe the bug
As part of a CD pipeline which uses Terraform to update EKS managed node groups with new customised base AMI's the below error is often encountered:
The issue is also encountered when manually running the Terraform
applyprocess to perform the EKS node updates from a local development environment/Expected Behavior
For the above exception, it is expected that a
RequestLimitExceededexception is raised which is considered a "retryable" error type and thus the AWS Go SDK is able to retry the API call subject to retry mode and maximum retry attempt parameters for the Terraform AWS provider.Current Behavior
The error encountered is:
This doesnt seem to be a valid error type to be retried according to aws/retry/retryable_error.go
Reproduction Steps
This error condition is frequently encountered when running a CD pipeline that invokes a Terraform
applyoperation to concurrently update a number of EKS managed node groups.This issue isn't directly reproducible via code implementation as the Terraform binary handles the relevant API calls.
Possible Solution
No response
Additional Information/Context
No response
AWS Go SDK V2 Module Versions Used
Compiler and Version used
Not applicable
Operating System and version
Linux x86_64, macOS Ventura 13.5.1