search

IntelLabs / vdms

VDMS: Your Favorite Visual Data Management System
MIT License
83 stars 31 forks source link

S3 Connection Layer Does Not Support Large Files #180

Open ifadams opened 1 month ago

ifadams commented 1 month ago

Issue: The basic AWS S3 connection interface doesn't support file uploads greater than a certain size. I believe the hard limit is 5GB, but AWS recommends multipart upload for any files larger than 100MB. Currently, attempts to upload large files may fail, with unpredictable results (i.e. client may think the upload succeeded, but it silently fails (only prints log message)).

Solution: Need to implement "multipart upload" for any file larger than a certain threshold.

ifadams commented 1 month ago

This may be a bigger problem than were making it out to be.