how to change the write-out buffer size?

[GoogleCodeExporter]

Which parameter should I play with (either by command line or
within the code) to change the size of the write-out buffer?
By default it flushes its output in 8 MB chunks and I might
need to lower that when I need to.

Thanks.

Original issue reported on code.google.com by seze...@gmail.com on 15 Dec 2007 at 9:13

[GoogleCodeExporter]

The -W flag changes the encode/decode window size, default is 8MB.  There is no
separate write-buffer parameter, but possibly it could be added.

Original comment by josh.mac...@gmail.com on 15 Dec 2007 at 6:40

• Changed state: Accepted

[GoogleCodeExporter]

The input winsize (-W) isn't the issue, my delta files
are already 1.4-1.8 MB. but it would be nice to have a
switch for the write buffer: Where should I look in the
source (if it's that easy) ?

Original comment by seze...@gmail.com on 15 Dec 2007 at 6:57

[GoogleCodeExporter]

The decode is allocating a buffer the same size as a single encoded window, 
which is
the size set by the encoder.  The buffer is allocated in xdelta3-decode.h:90, 
using
dec_tgtlen (the encoded window size).  So, reducing the encoder window size 
with -W
should reduce the decoder's write buffer.  It's difficult to decode a window of 
size
X w/ less than an X-byte buffer because a copy instruction can copy from 
anywhere in
the target window.

Original comment by josh.mac...@gmail.com on 15 Dec 2007 at 8:37

[GoogleCodeExporter]

I see: So it depends on how the delta file was created
and can't be changed that easily, right?

Original comment by seze...@gmail.com on 15 Dec 2007 at 8:43

[GoogleCodeExporter]

Changing the decoder to *allocate* less memory than the encoder used for a 
window is
difficult but not impossible.  Suppose your 8MB window, the decoder only 
allocates
2MB.  If it encounters a copy at 2.5MB into the window, referencing an offset of
1.5MB in the window, it now has to derive the data being copied using the 
stream of
instructions rather than the regenerated content.  It would  be simpler if the
encoder was restricted not to copy from the target (ie, self-referential 
compression,
as opposed to differencing), and there is a flag to turn off self-referential
compression but there is no way for the decoder to know that there are no
self-referential copies.  All things considered, it's best to just use a smaller
window in the encoder. It means worse compression because the encoder can't emit
self-referential copies outside the window.

Writing a function to *recode* the delta with a smaller window size is 
possible. In
fact, I'm working on a more general command to merge multiple deltas, the 
base-case
of which will be a command that allows changing the window size (after reading 
into
memory the entire delta).

Original comment by josh.mac...@gmail.com on 15 Dec 2007 at 8:55

[GoogleCodeExporter]

OK. (Sorry for bothering this much, I'm learning ;)
This isn't a problem for my case, at all, it seems:
I'm patching two files from a cdrom version of a
game and I just created new delta files with proper
-W arguments. All seems just fine.  Thanks!

Original comment by seze...@gmail.com on 15 Dec 2007 at 9:45

[GoogleCodeExporter]

Original comment by josh.mac...@gmail.com on 5 Jul 2008 at 1:30

• Changed state: Verified

[GoogleCodeExporter]

Original comment by josh.mac...@gmail.com on 28 Oct 2009 at 3:44

• Changed state: Fixed