Generate a set of test cases including automated deployment/test code around https PUT, S3 API calls to get the baseline performance from inside the AWS region and outside - using for example Azure to AWS
Optionally test for jitter, lag, error handling
Optionally test network optimization, use a C5 instance size (up to 25gbps in placement groups)
Test with/without mutipart handling via S3 Transfer Manager
Test with MTU, frame size, chunk size
Most of this testing simulates SFTP where we previously streamed or SCP'd the file/file(s) over
(1) S3 PUT via curl
223 Mbps throughput for up to 160 GB in max 5Gb pieces
In this example the instance was a burst T3a.large with 1 and 3.5Gb filesets already loaded up. So the use case was EC2 to S3 without optimizations
Generate a set of test cases including automated deployment/test code around https PUT, S3 API calls to get the baseline performance from inside the AWS region and outside - using for example Azure to AWS Optionally test for jitter, lag, error handling Optionally test network optimization, use a C5 instance size (up to 25gbps in placement groups) Test with/without mutipart handling via S3 Transfer Manager Test with MTU, frame size, chunk size
Most of this testing simulates SFTP where we previously streamed or SCP'd the file/file(s) over
(1) S3 PUT via curl 223 Mbps throughput for up to 160 GB in max 5Gb pieces
In this example the instance was a burst T3a.large with 1 and 3.5Gb filesets already loaded up. So the use case was EC2 to S3 without optimizations
from ec2 7 dec for 896mb = 900mbps
ubuntu@ip-172-31-91-213:~$ cat 20170821_eclipse_wyoming_glendo_15m_south_dark_only.mov 20170821_eclipse_wyoming_glendo_15m_south_dark_only.mov 20170821_eclipse_wyoming_glendo_15m_south_dark_only.mov 20170821_eclipse_wyoming_glendo_15m_south_dark_only.mov > large.mov
-rw-rw-r-- 1 ubuntu ubuntu 3757953512 Jun 27 20:47 large.mov 1649
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10885 ubuntu 20 0 186396 11168 9016 R 6.9 0.1 0:03.75 curl --request PUT --upload-file large.mov https://s3.amazonaws.com/lambda.input.useast1.packet.global/bacc2253eed74eaf9e5403fd69597bfa?AWSAcces+ 11746 root 20 0 639988 310804 24844 S 4.0 3.9 13381:09 kube-apiserver --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --anonymous-auth=false --bind-address=0.0.0.0 --runtime-config=authorizati+ 1 1651:15 135 sec for 3758Mb (3.5G) = 28MBps = 223Mbps
don't try over 5G -rw-rw-r-- 1 ubuntu ubuntu 10334372158 Jun 27 21:05 large_10g.mov
(2) s3 cp API ubuntu@ip-172-31-91-213:~$ aws s3 cp large_10g.mov s3://lambda.input.useast1.packet.global Completed 1.2 GiB/9.6 GiB (70.6 MiB/s) with 1 file(s) remaining
171410 completed 4.7 GiB/9.6 GiB (70.1 MiB/s) with 1 file(s) remaining
171630 140 sec for 10334Mb = 74MBps = 590Mbps