@ingalls I think we do need to implement the multi-threaded uploading for zip files >5 GB. I surveyed some users and they reported:
Across both islands (82 cams checked) I have 27 camera folders that are over 5GB, and out of those 17 are ~29GB, with one giant folder at 58GB.
So limiting the individual uploads to sub-5 GB zip files would put a fairly significant burden on the user.
Context
Single S3 uploads via presigned URLs are limited to 5GB uploads, so in order to support larger files, we'll need to chunk the zip file and send each chunk to S3 via its own presigned URL and S3's Multipart Upload feature. A complete example can be found here (and an accompanying blog post can be found here).
I'm not sure we need to implement the transfer acceleration feature they use in that repo, but I'm open to it.
@ingalls I think we do need to implement the multi-threaded uploading for zip files >5 GB. I surveyed some users and they reported:
So limiting the individual uploads to sub-5 GB zip files would put a fairly significant burden on the user.
Context Single S3 uploads via presigned URLs are limited to 5GB uploads, so in order to support larger files, we'll need to chunk the zip file and send each chunk to S3 via its own presigned URL and S3's Multipart Upload feature. A complete example can be found here (and an accompanying blog post can be found here).
I'm not sure we need to implement the transfer acceleration feature they use in that repo, but I'm open to it.