Merge failure with "out of memory"

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Build xdelta3 32-bit binaries.

2. Create forward patches for a series of 4 or more particular set of files.
   e.g. test.psd.1 -> test.psd.2 = test.2.diff; test.psd.2 -> test.psd.3 = test.3.diff; test.psd.3 -> test.psd.4 = test.4.diff.

3. Problem occurs when merging 3 or more patches, especially when successive 
files are identical.
  e.g. xdelta3 merge -m test.2.diff -m test.3.diff test.4.diff test.4.patch

What is the expected output? What do you see instead?
- Expected output is a merged patch (test.4.patch) which when applied to the 
1st file would give the last file.

- Instead, the following out of memory error appears:
xdelta3(548) malloc: *** mmap(size=1073725440) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
xdelta3: out of memory: Cannot allocate memory
xdelta3: further input required: Cannot allocate memory

What version of the product are you using? On what operating system?
- the error occurs in Xdelta3.0.0 and below (tested with 3.0z and 3.0y).
- On both Mac OS X 10.6 and Windows XP.

Please provide any additional information below.
A pair of test files that reproduce this bug:
1. "test.psd.1" attached.
2. "iPod_nano_chromatic.psd" available from: 
http://www.deviantart.com/download/111115606/iPod_nano_chromatic___Apple_by_zikk
zak.psd

Rename the above 2 files as follows in order to reproduce the situation 
described above:

cp test.psd.1 test.psd.2
mv iPod_nano_chromatic.psd test.psd.3
cp test.psd.3 test.psd.4

Original issue reported on code.google.com by sid.srin...@gmail.com on 8 May 2011 at 5:02

Attachments:

• test.psd.1

[GoogleCodeExporter]

The "out of memory" error is not an infinite loop as suspected above.

The 64-bit build on Mac OS X 10.6 works fine with the above test case, but uses 
excessive amounts of memory (> 1GB) even though the input patch sizes are 
fairly small (75 bytes, 20mb, 1.1kb, respectively).

So, it seems there needs to be some revamp of the merging algorithm in order to 
drastically reduce the memory footprint as I need it to work on 32-bit Win XP.

Original comment by sid.srin...@gmail.com on 10 May 2011 at 10:32

[GoogleCodeExporter]

Thanks for this interesting report. 
I'll have to investigate this later,
First, I have to re-install my Windows machine & build the latest release.

Original comment by josh.mac...@gmail.com on 18 Jun 2012 at 3:16

• Changed title: Merge failure with "out of memory"
• Changed state: Accepted

[GoogleCodeExporter]

Confirming under Linux as well. I cannot send a test case, because it has 
source binary file over 20Gb and one 500Mb diff. Still, the merge process uses 
8.8Gb of RAM on an x86-64 CentOS 6 host, and this is a bit overboard.

Original comment by aa.at...@gmail.com on 1 Feb 2013 at 7:08