S3: upload while reading input

livepeer / go-tools

Utility packages used across Livepeer Go repositories.

0 stars 2 forks source link

Closed AlexKordic closed 2 years ago

AlexKordic commented 2 years ago

In newfileUploadRequest() func we have read of entire input into memory io.Copy(part, fData) before starting HTTP request.

The fData is stdin in catalyst-uploader project.

Mist output binaries should read input at same pace as output is written. Caching entire input before starting output hurts Mist server workflow.

cyberj0g commented 2 years ago

AWS API doesn't support streaming upload directly
One way to achieve something close is to use multipart upload, but minimum partition size is 5Mb, which makes it impractical for small files