No automatic retry like AWS CLI

[per-oestergaard]

Description

There does not seem to be any automatic retry in the PowerShell implementation that AWS CLI has (https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-retries.html). I assume this must be an error as normally things are on par. Hence I have chosen a bug report. I apologize up front if that was a mistake.

Reproduction Steps

Anything that would return RequestLimitExceeded

Logs

n/a

Environment

As I could not find any reference to the exception or the environment variables used by AWS CLI, I assume it is everywhere.

Resolution

Perhaps implement retry logic maybe somewhere in CallAWSServiceOperation (unfortunately, I cannot grasp the full stack). Make the value settable with Set-AwsSomething or use the environment variables AWS CLI uses. AWS CLI seem to do it here in botocore: https://github.com/boto/botocore/blob/develop/botocore/endpoint.py#L136 and if there is a similar "hotspot" in the PowerShell implementation, if can be done there.

This is a :bug: bug-report

[ashishdhingra]

Hi @perost-l14,

Looks like AWS CLI has custom implementation for automatic retries using environment variables AWS_RETRY_MODE and AWS_MAX_ATTEMPTS. This appears to be a feature request for AWS Tools for Powershell, not a bug. As a workaround, you could try to implement your custom auto-retry operation until this feature if prioritized by development team. There is also Invoke-Retry cmdlet available in Microsoft TechNet script gallery that might be helpful.

Thanks, Ashish

[ashishdhingra]

Hi @perost-l14,

Good morning.

Upon discussion with development team, .NET SDK has support for retry mechanism and it should work in PowerShell (since it uses .NET SDK). While there is no formal PowerShell CmdLet argument, the retry mode and max attempts could be set by using the same set of environment variables as AWS CLI. Please refer to the following code in .NET SDK:

• https://github.com/aws/aws-sdk-net/blob/7c88eeceb52869892adb1fe50c3f06c8f1a6b126/sdk/src/Core/Amazon.Runtime/ClientConfig.cs#L253

• https://github.com/aws/aws-sdk-net/blob/7c88eeceb52869892adb1fe50c3f06c8f1a6b126/sdk/src/Core/Amazon.Runtime/ClientConfig.cs#L619

  For AWS_RETRY_MODE, the code appears to use RequestRetryMode enumeration.

The Retry Mode and Max Attempts, if set using environment variables, appear to be loaded in EnvironmentVariableInternalConfiguration class.

Please try using the environment variables for retry mode and max attempts, and let me know if it works. For failed scenario, you might want to examine requests using network monitoring tool (such as Fiddler).

Thanks, Ashish

[github-actions[bot]]

This issue has not recieved a response in 2 weeks. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

[per-oestergaard]

I haven't been able to force the retries, so I haven't been able to test this yet. Also, I have to easy way of detecting this from my existing code base as I cannot force it. If anybody has a way of forcing it, please share.

[per-oestergaard]

I did manage to force retries. However, changing the environment variables does not change the behavior. I have not been able to turn on logging as this is .net core so the instructions I find does not work. Any info would be appreciated.

[ashishdhingra]

Not sure if this works with the following PowerShell logic to create 50 EC2 instances (free tier eligible):

(Example taken from How do I avoid RequestLimitExceeded errors when programmatically launching multiple Amazon EC2 instances using PowerShell?)

• Keep Fiddler (network monitoring tool running)
• Execute the below statements to set the environment variables:

  $Env:AWS_RETRY_MODE="Standard"
  $Env:AWS_MAX_ATTEMPTS=4

• Execute the below script to create 50 EC2 instances:

  
  $params = @{
  ImageId='ami-09558250a3419e7d0'
  InstanceType = 't2.micro'
  AssociatePublicIp = $false
  MinCount = 10
  MaxCount=10
  }

for ($i=0;$i -lt 6;$i++){ $instance = New-EC2Instance @params }

This will throw the below exception:
```PowerShell
New-EC2Instance : You have requested more vCPU capacity than your current vCPU limit of 64 allows for the instance
bucket that the specified instance type belongs to. Please visit http://aws.amazon.com/contact-us/ec2-request to
request an adjustment to this limit.
At line:2 char:13
+ $instance = New-EC2Instance @params
+             ~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (Amazon.PowerShe...2InstanceCmdlet:NewEC2InstanceCmdlet) [New-EC2Insta
   nce], InvalidOperationException
    + FullyQualifiedErrorId : Amazon.EC2.AmazonEC2Exception,Amazon.PowerShell.Cmdlets.EC2.NewEC2InstanceCmdlet

New-EC2Instance : You have requested more vCPU capacity than your current vCPU limit of 64 allows for the instance
bucket that the specified instance type belongs to. Please visit http://aws.amazon.com/contact-us/ec2-request to
request an adjustment to this limit.
At line:2 char:13
+ $instance = New-EC2Instance @params
+             ~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (Amazon.PowerShe...2InstanceCmdlet:NewEC2InstanceCmdlet) [New-EC2Insta
   nce], InvalidOperationException
    + FullyQualifiedErrorId : Amazon.EC2.AmazonEC2Exception,Amazon.PowerShell.Cmdlets.EC2.NewEC2InstanceCmdlet

There are no 4 retry attempts fired to re-create remaining EC2 instances.

Not sure if something else needs to be configured or if the retry mechanism is applicable for Get or List CmdLets.

[ashishdhingra]

Refer How do I avoid RequestLimitExceeded errors when programmatically launching multiple Amazon EC2 instances using PowerShell? for workaround using delayed retry mechanism. Not sure if .NET SDK's retry mechanism handles RequestLimitExceeded errors (and if it should handle retry of these type of errors).

[ashishdhingra]

Hi @per-oestergaard,

Good morning.

Just wanted to add more inputs to this issue. AWS SDK for .NET (and hence PowerShell) does have RetryHandler for failed request. The default retry attempts is 5. However, what we are not sure if this retry handler should handle RequestLimitExceeded errors. Are you able to confirm if your logic if used in AWS CLI handles RequestLimitExceeded error? Also refer How do I avoid RequestLimitExceeded errors when programmatically launching multiple Amazon EC2 instances using PowerShell? for workaround using delayed retry mechanism.

Thanks, Ashish

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.