Memory problems: SOLVED.

[wvmarle]

Believe it or not, a single = sign completely solves the excessive memory use. I'm very surprised by this result. No need for memoryview so fully python 2.6 friendly. Proper code:

glaciercorecalls.py:324 while self.buffer_size >= self.part_size:

I tested it with a 128 MB chunk and the old code NOT using memoryview, and the results are stunning. Total memory use was: 640 MB With memoryview: 512 MB With this fix, and the orginal string-based code: 128 MB.

This works as long as the writer gets fed the correct block size, so when a complete block is in the buffer it can send out this complete block. Smart python will continue to refer copies of this buffer to the same memory location, and thus not use any extra memory.

Original situation:

• read block of data from file, put it in a string.
• call writer, put string in buffer (as list member; not copying data).
• buffer not larger than block size, don't start sending.
• read second block of data from file (2 blocks in memory)
• call writer, put string in buffer (as list member; not copying data).
• call send_part
• copy buffer into memoryview object (4 blocks in memory now)
• strip buffer down (3 blocks in memory)
• send out the block. Using the string buffer instead of memoryview makes it worse, total five block sizes in memory.

New situation:

• read block of data from file, put it in a string.
• call writer, put string in buffer (as list member; not copying data).
• buffer equals block size, start sending.
• copy buffer[] into string buf, identical data as there is just one member in the list, so points to the same memory block. No extra memory needed.
• reset buffer[] to empty state; there is no remainder.
• send out the block.

Wow, what a difference a single character can make!

For this analyses I used heapy to get memory use at various points in the program (use the svn trunk for python 2.7).

[offlinehacker]

Thanks, what about pull request? I still think memoryview should be there in case of python 2.7

[wvmarle]

I have open pull requests; github doesn't let me create one.

Memoryview or array suddenly doesn't make a difference as in either case there is one block in memory..

[uskudnik]

Hmm... didn't know github only lets you create only one... I'll try to merge your existing one ASAP.

[wvmarle]

Merge done; time to close this one.