Closed danopia closed 1 year ago
@danopia I was able to pull that helper in and it works, nicely done! I am going to run it through some load to see if there are any noticeable performance improvements.
Excellent, thanks for the validation effort ❤️
In my quick tests I noticed that memory usage isn't excellent. It seems that aws-api makes a copy or two of each request payload, so the default 20mb buffer (5mb*4) can actually become 60mb or so. That's not related to this helper, so I'll probably look at optimizing memory in a different PR.
@danopia performance did seem to improve, in particular for larger files approaching the 1-2 MB size. Though that would make sense. Pretty sweet!
Memory usage did seem to increase, but CPU increased more in the tests that I ran.
This was running on a container on Fargate with 1 vCPU and 2048 GB of memory, fwiw.
I'll merge this in the coming days (just a couple more unit tests) and then also verify with some of the other S3 providers before tagging a release.
Note that the function is now named managedUpload
to match AWS SDK, because it only multi-parts on files larger than 5MB so the name was misleading in the end.
Comparable in outcome to the SDK's AWS.S3.ManagedUpload class.
Fixes #30
Example
Remaining work