search

ised-isde-canada / cbmdsp-cp-sp-overall-architecture

1 stars 1 forks source link

Task: Test Portal type 3 (SFTP client) for maximum 1+TB S3 upload - priority 3 #20

Open obriensystems opened 3 years ago

obriensystems commented 3 years ago

Note: this is the 3rd priority after the thick(python) and thin(html/js) clients are done AWS Transfer has a $cost that will need to be scoped

BrianFanning commented 3 years ago

I've been trying to push the limits on how much bandwidth the SFTP server can handle. I tested an SFTP transfer from a Windows instance in the Oregon region to a managed SFTP server in Montreal region. Achieved upload speed of approximately 28 MiB/s (uploaded 1GB file in 42 seconds).

This page suggests 68MB/s+ speeds are possible if your upload speed is good enough

BrianFanning commented 3 years ago

I performed another test from us-west-2 EC2 instance to ca-central-1 SFTP server comparing transfer of one large file (1GB) vs. lots of smaller files (1000 x 1MB files):

Single 1GB file:

1000 1MB files:

Due to the additional overhead of uploading small files, it's approximately 5 times slower to upload individual files compared to compressing into a single file / zipping. Compression of course will make the overall file size smaller, so actually the performance benefits of using one single file are likely more than 5x

obriensystems commented 3 years ago

Brian, very nice, good news - an optional strategy for clients that need it around pre/post compression.

I should also revisit the SDK and run a 96 core spot VM C6... for a t3a.large standard VM we can see 233mbps for S3 PUT and 590mbps for S3 cli cp of just under 5G max of 4.7G http://wiki.obrienlabs.cloud/display/DEV/AWS+S3+Use+Cases#AWSS3UseCases-S3HTTPSPUTfromEC2 http://wiki.obrienlabs.cloud/display/DEV/AWS+S3+Use+Cases#AWSS3UseCases-S3CPAPIfromEC2