wvmarle
commented
11 years ago Well, this one is solved. But I really don't understand why it has to be done like that.
Key in the speed issue is line 258/265 where I commented out response.read() as I don't see the use of reading the response, and not doing anything with it. Well, it appears it does have a serious influence further down the line, as the next upload is so much faster!
Anyone with an idea on why this may happen like this?
No matter what going to make a patch adding these response.read() calls everywhere. It seems it doesn't take much time, and speeds up subsequent calls to the connection object significantly.
I noticed that recently my upload speeds dropped badly, from about 1.4 MB/s to 250-300 kB/s. So time for some investigation.
One of the reasons it's down is chunk size: when I increase from 1 MB to 64 MB I get back my old speeds, and more.
Another is this commit:
https://github.com/wvmarle/amazon-glacier-cmd-interface/commit/654928729f24be7d7e69cf427ae5ef4bb42ee50b
Before this commit I do 700-800 MB/s overall on 1MB chunks. After this commit it's down to <300 kB. I don't see what's the performance killer there. Any thoughts?
It also proves that chunk size is really important for upload performance, small chunks kill overall speed on faster connections, apparently starting the upload each time takes a long time