search

axman6 / amazonka-s3-streaming

Provides a conduit based interface to uploading data to S3 using the Multipart API
MIT License
20 stars 23 forks source link

Reuse buffer in streamUpload #35

Closed axman6 closed 1 year ago

axman6 commented 1 year ago

Implements the idea from #34 - I've done some basic testing using the s3upload executable and dd set to use a block size of 10MB and the upload appears work as expected.

axman6 commented 1 year ago

I've changed the implementation to now use a single buffer for the whole upload. In the future it might be nice to have multiple chunks uploading at the same time in parallel, which could be built around multiple reusable buffers and upload threads, but for now I think this is a good option that should keep memory usage at a minimum.

As the code now directly writes each ByteString to the buffer, instead of building up a Builder, each ByteString can be deallocated by the GC as soon as we're done with it.