When uploading lots of files using ObjectStorage.Insert(), each file allocates his own buffer. This results in increased CPU usage and high RSS memory.
Graph below shows how many space was allocated during upload of ~30k of 10MB files:
User should be able to pass his own buffer and be responsible for allocations or library shouldn't be so wasteful.
Pull request which adds user facing insert option is here: #632, I was asked to an open issue to discuss it.
cc. @tritone @codyoss
codyoss
commented
4 years ago
When uploading lots of files using ObjectStorage.Insert(), each file allocates his own buffer. This results in increased CPU usage and high RSS memory.
Graph below shows how many space was allocated during upload of ~30k of 10MB files:
User should be able to pass his own buffer and be responsible for allocations or library shouldn't be so wasteful.
Pull request which adds user facing insert option is here: #632, I was asked to an open issue to discuss it. cc. @tritone @codyoss