Memory requirement too high for large files?

[GoogleCodeExporter]

Just started testing your program on some 1 GB files.  Seems to work 
great. 

However, I was suprised by the following statement I found in the 
paper "Optimizing File Replication over Limited-Bandwidth Networks using 
Remote Differential Compression" 
(ftp://ftp.research.microsoft.com/pub/tr/TR-2006-157.pdf)

>Note that xdelta uses significantly
more memory than RDC (8 times the total file sizes for
xdelta, vs. less than 5% of the file size for RDC if the
signatures are kept in memory).<

None of the switches described here 
http://code.google.com/p/xdelta/wiki/TuningMemoryBudget lead me to believe 
that I'll need 8 GB to process a 1 GB file.  And, I do not have 8 GB on my 
laptop, and the program was quick (3 minutes) so I doubt a lot of paging 
is going on.

Please iluminate.  I'll do additional benchmarking if you need it.

Original issue reported on code.google.com by andyr...@gmail.com on 30 Jan 2008 at 10:16

[GoogleCodeExporter]

BTW, they use your program as the standard for compressibility. Nice.

Original comment by andyr...@gmail.com on 30 Jan 2008 at 10:22

[GoogleCodeExporter]

They used xdelta1.x for their comparison.
The new version was a complete rewrite, and the primary goal was not using 
linear
space.  So I'm pleased to report, this is a non-issue.

Original comment by josh.mac...@gmail.com on 30 Jan 2008 at 11:13

• Changed state: Verified

[GoogleCodeExporter]

xdelta3 lets you specify exactly how much memory you have, and should still
outperform xdelta1.  please let me know if it does not help.

Original comment by josh.mac...@gmail.com on 28 Oct 2009 at 3:34

• Changed state: Fixed