Dynamically adjust timeout for copy in EBS Copy Snapshot to S3

[ramo-j]

Currently the copy snapshot to S3 operation uses an exponential backoff in checking if the copy operation has completed. It hard fails when a limit is reached.

We should identify the size of the snapshot and use the AWS data transfer rates (need to be calculated) to better identify the waiting period.

[ramo-j]

Checking transfer times with the following, on an ap-se-2 m4.xl:

#!/usr/bin/bash

aws ec2 describe-regions --region ap-southeast-2 | jq -r '.Regions[].RegionName' | sort | while read region
do
    aws s3api create-bucket --bucket ramoj-test-bucket-$region --region $region --create-bucket-configuration LocationConstraint=$region > /dev/null
    echo $region
    # Get a few runs to show anomalies
    /usr/bin/time -f '%E' aws s3 cp --quiet /home/ec2-user/tmp/one-gig.bin s3://ramoj-test-bucket-$region/
    /usr/bin/time -f '%E' aws s3 cp --quiet /home/ec2-user/tmp/one-gig.bin s3://ramoj-test-bucket-$region/
    /usr/bin/time -f '%E' aws s3 cp --quiet /home/ec2-user/tmp/one-gig.bin s3://ramoj-test-bucket-$region/
    /usr/bin/time -f '%E' aws s3 cp --quiet /home/ec2-user/tmp/one-gig.bin s3://ramoj-test-bucket-$region/
    /usr/bin/time -f '%E' aws s3 cp --quiet /home/ec2-user/tmp/one-gig.bin s3://ramoj-test-bucket-$region/
    aws s3 rm --quiet s3://ramoj-test-bucket-$region/one-gig.bin
    aws s3api delete-bucket --bucket ramoj-test-bucket-$region --region $region
    echo
done

[ramo-j]

$ ./script.sh
ap-northeast-1
0:30.13
0:31.18
0:27.26
0:26.07
0:27.73

ap-northeast-2
0:27.98
0:31.59
0:28.97
0:29.82
0:29.48

ap-northeast-3
0:26.37
0:25.12
0:26.27
0:24.51
0:26.51

ap-south-1
0:32.89
0:30.02
0:28.92
0:30.07
0:29.94

ap-southeast-1
0:20.04
0:21.21
0:22.65
0:24.21
0:21.26

ap-southeast-2
0:11.84
0:11.74
0:11.68
0:11.71
0:11.89

ca-central-1
0:36.40
0:33.63
0:32.48
0:33.49
0:33.39

eu-central-1
0:40.40
0:40.47
0:43.77
0:44.42
0:44.86

eu-north-1
0:43.06
0:40.75
0:41.55
0:43.96
0:39.80

eu-west-1
0:40.35
0:38.00
0:40.46
0:38.17
0:39.18

eu-west-2
0:40.92
0:40.37
0:40.53
0:41.08
0:37.30

eu-west-3
0:41.36
0:42.21
0:43.32
0:39.60
0:41.90

sa-east-1
0:42.57
0:39.81
0:44.76
0:42.02
0:41.61

An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation: The specified location-constraint is not valid
us-east-1
0:01.39
0:01.39
0:01.39
0:01.39
0:01.34

An error occurred (NoSuchBucket) when calling the DeleteBucket operation: The specified bucket does not exist

us-east-2
0:31.74
0:34.64
0:36.99
0:32.78
0:32.39

us-west-1
0:28.52
0:28.53
0:27.28
0:30.93
0:30.47

us-west-2
0:28.63
0:30.74
0:26.47
0:27.78
0:28.72

~45 peak, so we'll round up to 60 to be conservative I reckon.