retries not always usefull

[VadimSafonov]

Describe the bug

when sdk client created with "retries" options and we do restoreObject with Tier=expedited and request was completed with error "Glacier expedited retrievals are currently not available, please try again later" sdk will retries this request

Expected Behavior

sdk not retries request on "Glacier expedited retrievals are currently not available, please try again later" and other special errors RestoreAlreadyInProgress

Current Behavior

sdk do retries

Reproduction Steps

1 create sdk client with retries 2 enable cloudtrail for bucket (to see logs) 3 do restoreObject

Possible Solution

No response

Additional Information/Context

No response

SDK version used

3.309.0

Environment details (Version of PHP (php -v)? OS name and version, etc.)

PHP 7.2.34 (cli) (built: Jan 28 2022 10:18:52) ( ZTS )

[yenfryherrerafeliz]

Hi @VadimSafonov, sorry to hear about your issues. Would you please be able to provide debug logs here so we can better assist you. You can enable debug logs by passing the flag debug set to true to the client. Here is a simple example:

$client = new S3Client([
    'region' => 'us-east-2',
    'debug' => true
]);

Please make sure you redact any sensitive information.

Thanks!

[VadimSafonov]

Hello, see log in the attach. SDK was updated to the latest version 3.316.5 Client was create with params

$s3 = new Aws\S3\S3Client([
    "version" => "latest",
    "region"  => "us-east-1",
    "retries" => 5,
    "use_accelerate_endpoint" => true,
    "credentials" => [
        "key"    => "KEY",
        "secret" => "SECRET",
    ],
    "debug" => true
]);

restore.log

[VadimSafonov]

if required i can do test case with enabled cloudtrail logs

[RanVaknin]

Hi @VadimSafonov ,

I looked at your logs and I can see that the SDK retried the request 6 times. The SDK will decide on which behavior to retry based on the http status code returned in the response. In this case the service returned a 503, which is a throttling error, so the SDK will retry request.

If you need to wait for longer between each retry, you need to override the default retryer and provide your own backoff logic to fit this operation. Our retryer is generic and is bound to the I/O of the interaction with the service. We don't maintain a custom retry strategy on a per operation basis, so if restoreObject is not ready within a reasonable amount of time, you might want to implement a waiter that polls for a valid response if you rely on the results from this restoreObject call.

Thanks, Ran~

[VadimSafonov]

I want to have one instance of client with configured retries, but in case of custom errors like RestoreAlreadyInProgress and GlacierExpeditedRetrievalNotAvailable I don't want that the client to use attempts and stop after first error. Is it possible?

[yenfryherrerafeliz]

Hi @VadimSafonov, since we do not have a straight forward way to provide custom retry deciders, we may have an option for doing this through middlewares. I will get back to you soon with a simple example for doing it on this way.

Thanks!

[yenfryherrerafeliz]

Hi @VadimSafonov, here is one approach for overriding our default retry decider implementation:

?php
require '../vendor/autoload.php';

use Aws\AwsClient;
use Aws\CommandInterface;
use \Aws\Retry\Configuration as RetryConfiguration;
use \Aws\Retry\QuotaManager as RetryQuotaManager;
use Aws\RetryMiddlewareV2;
use Aws\S3\S3Client;
use \Aws\Exception\AwsException;

final class AwsRetryUtils {
    private function __construct() {}

    public static function applyCustomRetryDecider(AwsClient &$client)
    {
        // Please replace the first parameter with the desired mode: standard | adaptive | legacy
        // Please replace the second parameter with the desired max attempts.
        // By default, maxAttempts is set to 3, which means you can omit providing it if is what you desired.
        $config = new RetryConfiguration('standard', 5);
        $defaultDecider = RetryMiddlewareV2::createDefaultDecider(
            new RetryQuotaManager(),
            $config->getMaxAttempts()
        );
        // Here, I just create a custom decider on top of the default decider.
        // Something to be aware of is that the default decider is the one that takes into
        // account the max attempts constraint, among other rules such as quota management,
        // backoff, etc. which means that if you decide to ignore the custom decider then, it is
        // at your own responsibility. Also, if you create a rule for retrying then, you need
        // to make sure you also validate the number of attempts is not over the maxAttempts,
        // otherwise you will hang in an infinitive loop of retries.
        $customDecider = function ($attempts, CommandInterface $cmd, $result) use ($defaultDecider) {
            // Here, you can add your custom condition for what you do not want to retry.
            // $result can be an instance of ResultInterface, AwsException, and Throwable.
            // If you just want to validate the status code, you can do something like this:
            if ($result instanceof AwsException &&  $result->getStatusCode() === 503) {
                return false; // or true
            }

            // If you want to validate the status code and the message, you can do something like this:
            // Please replace YOUR_DESIRED_PATTERN with the desired pattern.
            // You can use regular expressions as well.
           /* if ($result instanceof AwsException
                &&  $result->getStatusCode() === 503
                && preg_match('YOUR_DESIRED_PATTERN', $result->getMessage())) {
                return false; // or true
            } */

            return $defaultDecider($attempts, $cmd, $result);
        };

        $retryOptions = [
            'decider' => $customDecider,
        ];
        $client->getHandlerList()->remove('retry');
        $client->getHandlerList()->appendSign(
            RetryMiddlewareV2::wrap($config, $retryOptions),
            'retry'
        );
    }
}

// CREATE THE CLIENT
$client = new S3Client([
    'region' => 'us-east-1',
    'debug' => false
]);
// APPLY A CUSTOM RETRY DECIDER TO THE CLIENT
AwsRetryUtils::applyCustomRetryDecider($client);
// DO THE OPERATIONS
$result = $client->listObjects([
    'Bucket' => 'BUCKET_NAME'
]);

Please let me know if that helps or if you have any questions.

Thanks!

[github-actions[bot]]

This issue has not recieved a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.