Open obriensystems opened 3 years ago
I've been trying to push the limits on how much bandwidth the SFTP server can handle. I tested an SFTP transfer from a Windows instance in the Oregon region to a managed SFTP server in Montreal region. Achieved upload speed of approximately 28 MiB/s (uploaded 1GB file in 42 seconds).
This page suggests 68MB/s+ speeds are possible if your upload speed is good enough
I performed another test from us-west-2 EC2 instance to ca-central-1 SFTP server comparing transfer of one large file (1GB) vs. lots of smaller files (1000 x 1MB files):
Single 1GB file:
1000 1MB files:
Due to the additional overhead of uploading small files, it's approximately 5 times slower to upload individual files compared to compressing into a single file / zipping. Compression of course will make the overall file size smaller, so actually the performance benefits of using one single file are likely more than 5x
Brian, very nice, good news - an optional strategy for clients that need it around pre/post compression.
I should also revisit the SDK and run a 96 core spot VM C6... for a t3a.large standard VM we can see 233mbps for S3 PUT and 590mbps for S3 cli cp of just under 5G max of 4.7G http://wiki.obrienlabs.cloud/display/DEV/AWS+S3+Use+Cases#AWSS3UseCases-S3HTTPSPUTfromEC2 http://wiki.obrienlabs.cloud/display/DEV/AWS+S3+Use+Cases#AWSS3UseCases-S3CPAPIfromEC2
Note: this is the 3rd priority after the thick(python) and thin(html/js) clients are done AWS Transfer has a $cost that will need to be scoped