search

esmero / strawberryfield

A Field of strawberries
GNU Lesser General Public License v3.0
10 stars 5 forks source link

ISSUE-286: Break the 5 Gbytes barrier without breaking the bank #288

Closed DiegoPino closed 8 months ago

DiegoPino commented 9 months ago

See #286

This adds a new config form element:

image

And enforces that setting (5 Gbytes by default, the exact default we need) to attempt either a MultiPart Upload Operation (with a multipart retry) or a MultiPart Copy Operation (between S3 URIs) to manage very large files (basically anything larger to 5 Gbytes if you don't change the default settings.)

Tested and re-retested but might YET again test tomorrow before merging.

NOTE: Multipart upload DOES NOT WORK if MIN.IO is routing AWS S3. MIN.IO passes the wrong headers breaking the AWS S3 Client/response. It works with either S3 directly OR with MIN.IO directly serving local storage via the S3 API.

DiegoPino commented 8 months ago

@alliomeria I need to merge this one. I will do yet around round of tests by deleting/running an AMI set on my localhost and then also trying a manual ingest.