Closed ramo-j closed 3 years ago
Checking transfer times with the following, on an ap-se-2 m4.xl:
#!/usr/bin/bash
aws ec2 describe-regions --region ap-southeast-2 | jq -r '.Regions[].RegionName' | sort | while read region
do
aws s3api create-bucket --bucket ramoj-test-bucket-$region --region $region --create-bucket-configuration LocationConstraint=$region > /dev/null
echo $region
# Get a few runs to show anomalies
/usr/bin/time -f '%E' aws s3 cp --quiet /home/ec2-user/tmp/one-gig.bin s3://ramoj-test-bucket-$region/
/usr/bin/time -f '%E' aws s3 cp --quiet /home/ec2-user/tmp/one-gig.bin s3://ramoj-test-bucket-$region/
/usr/bin/time -f '%E' aws s3 cp --quiet /home/ec2-user/tmp/one-gig.bin s3://ramoj-test-bucket-$region/
/usr/bin/time -f '%E' aws s3 cp --quiet /home/ec2-user/tmp/one-gig.bin s3://ramoj-test-bucket-$region/
/usr/bin/time -f '%E' aws s3 cp --quiet /home/ec2-user/tmp/one-gig.bin s3://ramoj-test-bucket-$region/
aws s3 rm --quiet s3://ramoj-test-bucket-$region/one-gig.bin
aws s3api delete-bucket --bucket ramoj-test-bucket-$region --region $region
echo
done
$ ./script.sh
ap-northeast-1
0:30.13
0:31.18
0:27.26
0:26.07
0:27.73
ap-northeast-2
0:27.98
0:31.59
0:28.97
0:29.82
0:29.48
ap-northeast-3
0:26.37
0:25.12
0:26.27
0:24.51
0:26.51
ap-south-1
0:32.89
0:30.02
0:28.92
0:30.07
0:29.94
ap-southeast-1
0:20.04
0:21.21
0:22.65
0:24.21
0:21.26
ap-southeast-2
0:11.84
0:11.74
0:11.68
0:11.71
0:11.89
ca-central-1
0:36.40
0:33.63
0:32.48
0:33.49
0:33.39
eu-central-1
0:40.40
0:40.47
0:43.77
0:44.42
0:44.86
eu-north-1
0:43.06
0:40.75
0:41.55
0:43.96
0:39.80
eu-west-1
0:40.35
0:38.00
0:40.46
0:38.17
0:39.18
eu-west-2
0:40.92
0:40.37
0:40.53
0:41.08
0:37.30
eu-west-3
0:41.36
0:42.21
0:43.32
0:39.60
0:41.90
sa-east-1
0:42.57
0:39.81
0:44.76
0:42.02
0:41.61
An error occurred (InvalidLocationConstraint) when calling the CreateBucket operation: The specified location-constraint is not valid
us-east-1
0:01.39
0:01.39
0:01.39
0:01.39
0:01.34
An error occurred (NoSuchBucket) when calling the DeleteBucket operation: The specified bucket does not exist
us-east-2
0:31.74
0:34.64
0:36.99
0:32.78
0:32.39
us-west-1
0:28.52
0:28.53
0:27.28
0:30.93
0:30.47
us-west-2
0:28.63
0:30.74
0:26.47
0:27.78
0:28.72
~45 peak, so we'll round up to 60 to be conservative I reckon.
Currently the copy snapshot to S3 operation uses an exponential backoff in checking if the copy operation has completed. It hard fails when a limit is reached.
We should identify the size of the snapshot and use the AWS data transfer rates (need to be calculated) to better identify the waiting period.