Closed jukkaho closed 12 years ago
Interesting. I think the culprit was the misalignment detector. Can you try the version I've just pushed?
That is much better. Memory use in my case was down to 184 MB when using ~50 GB temporary file. With less things to track, it was much faster as well.
Memory use gets really high when trimming RAID with two SSD drives and a lot of free space. Python also crashed with some internal double free errors when trying to trim using a file as large as 40 GB. I have successfully tested the program up to 30 GB temporary file sizes and it needed about 4 GB of RAM.
I was able to work around this by removing temporary file deletion from the code and using several ~10 GB files sequentially until I run out of disk space. Then I just removed all the trimmed temporary files at the end. This seemed to restore my write speeds to original performance level.
Additional benefit of using several smaller files is that the disk is almost full for shorter amount of time as checking of the last piece takes less time now.