biigle / user-storage

:m: BIIGLE module to offer file upload and storage for users
GNU General Public License v3.0
0 stars 0 forks source link

Improve upload performance #3

Open mzur opened 2 years ago

mzur commented 2 years ago

We should use multiple concurrent uploads if possible. It could be done like this: Track both the cumulated size and number of the files that are uploading at a given moment. If the size and number is below a given threshold (e.g. 50 MB and 5 files), immediately start uploading the next file. If not, wait for an upload to finish and check again. This way, concurrent uploads can be used to upload many smaller files but large files are uploaded one after the other.

mzur commented 2 years ago

This must take chunked files into account (see #9). Concurrent uploads should only be used when no chunked file is uploaded. This can probalby be solved by setting the concurrent upload size threshold (e.g. 50 MB) below the chunk size (e.g. 100 MB).