search

lz4 / lz4-java

LZ4 compression for Java
Apache License 2.0
1.1k stars 253 forks source link

Compressed message doesn't start with magic number? #36

Closed tarelli closed 9 years ago

tarelli commented 10 years ago

Hi, I am trying to use this library, here is the simple class I am using to wrap it.

https://github.com/openworm/org.geppetto.core/blob/181813152b1747428a2c1aa3126dc3eb40638bb8/src/main/java/org/geppetto/core/common/LZ4Compress.java

When I try to compress the following string:

{"type":"read_url_parameters","data":"{}"}

the result of the compress call is the following:

[-16, 27, 123, 34, 116, 121, 112, 101, 34, 58, 34, 114, 101, 97, 100, 95, 117, 114, 108, 95, 112, 97, 114, 97, 109, 101, 116, 101, 114, 115, 34, 44, 34, 100, 97, 116, 97, 34, 58, 34, 123, 125, 34, 125]

Shouldn't this start with the magic number 0x184D2204? I am trying to decode this with this library and it complains with:

Invalid magic number: 227B1BF0 @0

Am I doing something wrong? Thanks!

ecki commented 10 years ago

The Compressor#compress methods which you are using create raw blocks, and not the LZ4 stream format (as specified in http://fastcompression.blogspot.fr/2013/04/lz4-streaming-format-final.html with a Magic number of 0x184D2204 (LE)).

There is a stream format wrapper in lz4-java (LZ4OutputStream), however it looks like this one uses its own (homegrown or older?) format. It uses a Magic string: "LZ4Block".

for shorter strings it might be best to not use the streaming format to avoid the overhead, but then your receiving application need to be able to handle the raw blocks. (And you might want to add your own framing and checksums).

jpountz commented 9 years ago

This is correct. The streaming format attempt that is in lz4-java started before the upstream project defined a streaming format (see https://github.com/jpountz/lz4-java/issues/21), so it's both homegrown and older. :-)