search

alexmojaki / s3-stream-upload

Manages streaming of data to AWS S3 without knowing the size beforehand and without keeping it all in memory or writing to disk.
MIT License
208 stars 62 forks source link

Memory consuming #2

Closed minhtrex closed 7 years ago

minhtrex commented 8 years ago

I use the sample code of StreamTransferManager with a little modify in stream generator for bigger data.

I try to upload 1GB to S3. While the program is running, it consumes about 700MB.

Did i make something wrong? Or it's just a normal behavior of the library?

Thanks in advance. And sorry for my bad English

alexmojaki commented 7 years ago

Sorry for only getting to this now, I seem to have missed it from my notifications.

I managed to reproduce this behavior and I don't think it's related to the library, it seems to be Java allocating lots of memory just in case by default. Adding the option -Xmx50m (Max heap size = 50 MB) to the java run command makes the whole process take only about 100 MB.

However in the process of looking at this I did notice that there was a dependency that's only meant to be for tests, so if you upgrade to 1.0.1 you should get a small improvement in memory usage from there as well. But mostly do the heap size configuration.