This PR completes a naive / strawman implementation of multipart upload.
Main components:
Uploader - implements multipart upload
Takes the input stream and calculates part size using the size hint, then creates N workers that simply loop until no more parts from the part reader are available. Returns an UploadHandle to caller after the MPU is started.
UploadHandle - returned to caller to control an upload (e.g. pause/resume/progress/wait for completion). Currently CompleteMultipartUpload is invoked by "joining" the handle. Joining involves waiting for all of the parts to be uploaded and then completing the call.
ConcurrencySetting/TargetPartSize - introduced and refactored new common settings for expressing concurrency and part size settings that allow for an "auto" mode as the default. Right now these just default to some hard coded constant but in future leaves us room to introduce whatever logic we want here.
Future/Questions
Checksums are ignored right now, we need a design around how we want checksums to work for both upload/download. I've tried to place TODO/FIXME comments where appropriate as it relates to this.
Errors definitely need reworked at this point. In particular I've placed todo/fixme around completing an upload as we may end up with one or more part failures and possible a failure related to aborting the upload as well. How this gets exposed to the user needs some thought. Most likely we want to move towards defining an UploadError struct with different error "kinds" that capture various states/composite failures.
Single part upload (content length < min part size) is not implemented yet.
More tests around various scenarios (e.g. failed part, failed abort during processing a failed part, etc). I found using our experimental mocks package a bit tedious to setup, wondering if this is the direction to go still or not or if we have criteria for when to leverage this vs other testing strategies.
Questions on when various upload request / response fields get set when splitting vs not splitting the request. Some of the fields only apply to one case or the other.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Description of changes:
This PR completes a naive / strawman implementation of multipart upload.
Main components:
UploadHandleto caller after the MPU is started.CompleteMultipartUploadis invoked by "joining" the handle. Joining involves waiting for all of the parts to be uploaded and then completing the call.Future/Questions
UploadErrorstruct with different error "kinds" that capture various states/composite failures.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.