search

egnyte / egnyte-js-sdk

Javascript SDK to work with Egnyte Public APIs
15 stars 16 forks source link

Chunked upload memory leak #8

Open luis-figueiredo opened 3 years ago

luis-figueiredo commented 3 years ago

The chunked upload functionality seems to cause a RSS memory leak in nodeJS. Heap memory is stable but RSS keeps growing.

I have implemented both chunked upload options: as stream through streamToChunks and manually chunking and sending them myself through startChunkedUpload, sendChunk and sendLastChunk.

Both versions work perfectly fine in terms of getting the file uploaded, but, for example, my rss memory for uploading a 350MB file grows up to 1GB with either version, which makes absolutely no sense.

I am using a chunksize of 10MB

Steps to reproduce: Upload a file through egnyte-js-sdk either as stream or manually chunking Log memory usage through global process object console.log('memory rss in MB: ', process.memoryUsage().rss / 1024 / 1024);

If you need anymore information please let me know so I can provide it. This is a problem for me since it causes my application to run out of memory when uploading very large files.

As a side note, I don't think this can be due to my implementation outside of egnyte-js-sdk, since I tried just for testing the same algorithm to upload to sharepoint online and memory usage is almost perfectly constant, no leakage.

naugtur commented 3 years ago

Growing rss only is not a leak really. If the heap size goes down, the memory is cleared. Rss grows to cover for the largest allocations and then node doesn't release it when heap shrinks so it doesn't have to allocate again. There's some logic to it.

We'll eventually release a brand new major version with a different http client and the behavior may be different.

chaoyangnz commented 3 years ago

I am wondering what is the benefit to use chunk uploading?