Support LZMA compression

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Try xdelta3 -s 1.lzma 2.lzma test.xd
2. Notice that test.xd is as big as 1.lzma

What is the expected output? What do you see instead?
test.xd should be little bigger than sizeof(2.lzma - 1.lzma )

What version of the product are you using? On what operating system?
latest SVN on Linux

Original issue reported on code.google.com by ismail.d...@gmail.com on 22 Oct 2006 at 6:42

[GoogleCodeExporter]

I need to know the magic number for lzma files and the command-line syntax for 
compressing and decompressing.

Original comment by josh.mac...@gmail.com on 24 Oct 2006 at 4:05

[GoogleCodeExporter]

First you need lzma-utils from http://tukaani.org/lzma/lzma-4.32.0beta3.tar.gz 
. Usual 
configure && make && make install apply.

Compressing the input :

lzmash input_file

Decompressing :

lzmash -d input_file.lzma

I don't about the magic number for LZMA.

Thanks,
ismail

Original comment by ismail.d...@gmail.com on 26 Oct 2006 at 11:44

[GoogleCodeExporter]

I mean I don't know about the magic number for LZMA, I guess header could be 
used to 
identify them.

Thanks,
ismail

Original comment by ismail.d...@gmail.com on 26 Oct 2006 at 11:45

[GoogleCodeExporter]

Any news on this one?

Original comment by ismail.d...@gmail.com on 15 Nov 2006 at 10:40

[GoogleCodeExporter]

In the source release you suggested, I found a magic file, with unfortunate 
news:

# It was decided to use raw LZMA stream as the file format of LZMA
# compressed files. This was an unfortunate decision since the files do
# not have any magic first bytes.

# LZMA files have no magic bytes. Add this to the very end of
# /etc/file/magic to prevent mismatches.
#
# Some assumptions are made that limit the matching:
#   - The LZMA header has 64-bit value field for uncompressed size;
#     these checks assume that the uncompressed size is less than
#     2^48 bytes (256 TiB). Fix this when files this big are common. ;-)
#   - Only dictionary sizes 2^n where 0 <= n <= 28 are recognized.
#   - File size should be checked, the minimum is 18 bytes.
#
# LZMA header structure (first 13 bytes):
#   0     LZMA properties, 0x00 - 0xE1
#   1-4   Dictionary size (little endian)
#   5-12  Uncompressed size, filled with 0xFF if unknown (streamed data)
#
# This file is in the public domain. You can do whatever you want with it.

What I'd like to do is add a commandline flag to set the external 
compress/decompress
routines.  Search for "lzma" in xdelta3-main.h, there's a line you can uncomment
which at least worked in my tests.

Original comment by dotdotis...@gmail.com on 10 Dec 2006 at 9:54

• Changed state: Accepted

[GoogleCodeExporter]

I uncommented the lzma line in xdelta3-main.h but I still seem to get bad 
results
with latest SVN:

[~]> ls -al *.lzma
-rw-r--r-- 1 cartman users 13195767 Ara 14 09:49 dest.lzma
-rw-r--r-- 1 cartman users 13121405 Ara 14 09:49 source.lzma

[~]> ~/SVN/xdelta3/xdelta3-everything -f -s source.lzma dest.lzma test.xdelta
xdelta3: warning: input position 262144 no source copies
xdelta3: warning: input position 524288 no source copies
xdelta3: warning: input position 786432 no source copies
xdelta3: warning: input position 1048576 no source copies
xdelta3: warning: input position 1310720 no source copies
xdelta3: warning: input position 1572864 no source copies
xdelta3: warning: input position 1835008 no source copies
xdelta3: warning: input position 2097152 no source copies
xdelta3: warning: input position 2359296 no source copies
xdelta3: warning: input position 2621440 no source copies
xdelta3: warning: input position 2883584 no source copies
xdelta3: warning: input position 3145728 no source copies
xdelta3: warning: input position 3407872 no source copies
xdelta3: warning: input position 3670016 no source copies
xdelta3: warning: input position 3932160 no source copies
xdelta3: warning: input position 4194304 no source copies
xdelta3: warning: input position 4456448 no source copies
xdelta3: warning: input position 4718592 no source copies
xdelta3: warning: input position 4980736 no source copies
xdelta3: warning: input position 5242880 no source copies
xdelta3: warning: input position 5505024 no source copies
xdelta3: warning: input position 5767168 no source copies
xdelta3: warning: input position 6029312 no source copies
xdelta3: warning: input position 6291456 no source copies
xdelta3: warning: input position 6553600 no source copies
xdelta3: warning: input position 6815744 no source copies
xdelta3: warning: input position 7077888 no source copies
xdelta3: warning: input position 7340032 no source copies
xdelta3: warning: input position 7602176 no source copies
xdelta3: warning: input position 7864320 no source copies
xdelta3: warning: input position 8126464 no source copies
xdelta3: warning: input position 8388608 no source copies
xdelta3: warning: input position 8650752 no source copies
xdelta3: warning: input position 8912896 no source copies
xdelta3: warning: input position 9175040 no source copies
xdelta3: warning: input position 9437184 no source copies
xdelta3: warning: input position 9699328 no source copies
xdelta3: warning: input position 9961472 no source copies
xdelta3: warning: input position 10223616 no source copies
xdelta3: warning: input position 10485760 no source copies
xdelta3: warning: input position 10747904 no source copies
xdelta3: warning: input position 11010048 no source copies
xdelta3: warning: input position 11272192 no source copies
xdelta3: warning: input position 11534336 no source copies
xdelta3: warning: input position 11796480 no source copies
xdelta3: warning: input position 12058624 no source copies
xdelta3: warning: input position 12320768 no source copies
xdelta3: warning: input position 12582912 no source copies
xdelta3: warning: input position 12845056 no source copies

[~]> ls -al test.xdelta
-rw-r--r-- 1 cartman users 13196817 Ara 14 09:57 test.xdelta

Original comment by ismail.d...@gmail.com on 14 Dec 2006 at 7:58

[GoogleCodeExporter]

Ignore my last message input files were zip files containing tar.lzma archives. 
I
will re-test again and report. Thanks.

Original comment by ismail.d...@gmail.com on 14 Dec 2006 at 10:00

[GoogleCodeExporter]

Ok testing with two tar.lzma archive resulted in a bigger delta compared to 
files
which is interesting. Are you interested on sample files?

Regards,
ismail

Original comment by ismail.d...@gmail.com on 15 Dec 2006 at 8:24

[GoogleCodeExporter]

Just to reiterate, this issue can't be solved without adding new command-line 
flags
to explicitely set the external-compression/decompression program and flags. 
However,
external compression is just a convenience and you could easily write a script 
to
invoke xdelta3 after decompressing the two files. I've written some notes on 
external
compression here: http://code.google.com/p/xdelta/wiki/ExternalCompression

Original comment by josh.mac...@gmail.com on 16 Jan 2007 at 4:41

• Changed state: WontFix

[GoogleCodeExporter]

*sigh* I think at this point, everyone - probably including the original 
authors -
REALLY wish they hadn't done that...

But, it's not strictly true that there's no way to identify an LZMA stream.  
It's not
"straightforward", but I know I've seen it.  If you'd be willing, I'd certainly 
be
happy to go find that algorithm.

Original comment by ElecC...@gmail.com on 10 Oct 2009 at 4:51

[GoogleCodeExporter]

AFAIK lzma was deprecated and has been replaced by xz.
Also check out http://code.google.com/p/xdelta/issues/detail?id=87

Original comment by shinin...@gmail.com on 10 Oct 2009 at 4:56

[GoogleCodeExporter]

Supporting xz is a better option, LZMA is moving into that direction too.

Original comment by ismail.d...@gmail.com on 12 Oct 2009 at 8:23

[GoogleCodeExporter]

Xz is now supported (SVN 298)

Original comment by josh.mac...@gmail.com on 28 Oct 2009 at 3:45

• Changed state: Fixed