Download - Make one big request for all S3 links instead of one for each file

glassypdm / glassypdm-client

CAD-agnostic PDM package

https://glassypdm.org/

GNU General Public License v3.0

1 stars 1 forks source link

Download - Make one big request for all S3 links instead of one for each file #11

Closed joshtenorio closed 11 months ago

joshtenorio commented 12 months ago

Currently we make a GET request for each file to get its S3 link; If we make one big request for all the S3 links we need we should be able to see some sort of performance increase. This will require a change on the server side too.

[x] measure time elapsed while downloading files
[ ] supported change on server side
[ ] implement change, measure time elapsed

joshtenorio commented 11 months ago

Pre change:

Downloading 650ish files (637 MB): 833 seconds (server/something kinda maybe froze too?)
when testing with localhost, downloading 650ish files took about 507 seconds

Post change with big request:

with localhost, downloading 650ish files took 422 seconds (~16% decrease)

The main concern now is having a reasonably long enough timeout for the presigned url.

joshtenorio commented 11 months ago

This approach has some issues; replacing the for loop with an awaited map is faster across both server and localhost. e464eeca63ee49b32dd88abf14da49e6a6ae9105