Using lz4 without Node.js

[tarelli]

Is it possible to use this library in a non Node.js application (client-side)? Thanks

[tarelli]

Update: I am trying using decoder-js.js from the bin folder (I only need decompression). Given that I'm not in Node.js environment how would I use it (I have no buffers etc)? I attempted the following but for some reason the output is huge (30MB approx, few bytes expected):

                ...
            var output = [];
            var input = GEPPETTO.MessageSocket.string2Bin(msg.data);

            lz4.LZ4_uncompressChunk(input,output);

            var decompressedData=GEPPETTO.MessageSocket.bin2String(output);
                ...

        GEPPETTO.MessageSocket.string2Bin=function(str) {
              var result = [];
              for (var i = 0; i < str.length; i++) {
                result.push(str.charCodeAt(i).toString(2));
              }
              return result;
        };

        GEPPETTO.MessageSocket.bin2String=function(array) {
              var result = "";
              for (var i = 0; i < array.length; i++) {
                result += String.fromCharCode(parseInt(array[i], 2));
              }
              return result;
        };

Any suggestion would be extremely helpful!

[tarelli]

I now understand I have to use ArrayBuffer/Uint8Array still trying to understand how though, a snippet would be very helpful.

[pierrec]

Hello,

There is a version of the decoder written in JS. There isnt anything for the encoder as I ran into the JS limitation of 32 bits integers. I plan on looking into it again at some point but am lacking time.

I will look into your issue regarding the js decoder though.

Cheers,

Pierre

[pierrec]

Hello,

I have reviewed your issue and committed changes on branch 0.1 as the js decoder is not updated yet for the new LZ4 Stream format (the magic number still references the lz 4 previous format). I will eventually update it.

I have one simple example which requires a proper Buffer implementation such as the one you already tried ;) (https://github.com/arextar/browser-buffer).

Rgds,

Pierre

[pierrec]

I hope you managed to get it to work. Closing this issue.

[tarelli]

@pierrec do you have that sample you mention somewhere that I can take a look at? Thanks!

[pierrec]

Ah sorry, forgot to mention where the example is:

https://github.com/pierrec/node-lz4/blob/master/examples/www/uncompress.html

[tarelli]

@pierrec the line that does the decoding is commented. The method decode doesn't seem to exist anymore, I tried to use lz4.LZ4_uncompress instead but I get Error: Invalid data at 0.

The code I am using is the following

GEPPETTO.MessageSocket.bufferFromUTFString =function (str) {
              var bytes = []
                , tmp
                , ch;

              for(var i = 0, len = str.length; i < len; ++i) {
                ch = str.charCodeAt(i);
                if(ch & 0x80) {
                  tmp = encodeURIComponent(str.charAt(i)).substr(1).split('%');
                  for(var j = 0, jlen = tmp.length; j < jlen; ++j) {
                    bytes[bytes.length] = parseInt(tmp[j], 16);
                  }
                } else {
                  bytes[bytes.length] = ch ;
                }
              }

              return new Uint8Array(bytes);
            };

        GEPPETTO.MessageSocket.socket.onmessage = function(msg)
        {           
            var uint8buf=GEPPETTO.MessageSocket.bufferFromUTFString(msg.data);
            var inputBuffer = new Buffer(uint8buf,"UTF8");

            var decompressedData = lz4.LZ4_uncompress(inputBuffer);
            var parsedServerMessage = JSON.parse(decompressedData);

Can you spot anything wrong? Is the error I am getting the result of the library not being updated to the latest LZ4 format (I'm using this to compress the message server side)? Also when you say you will eventually update it do you know when you will have the time? This would be very useful! Thanks.

[pierrec]

Hello,

Sorry about the wrong html file... I pointed you at the current master file which does not work because it can only decode data encoded with the previous version of lz4 archive format. I think the lz4 compression library you have uses the 1.4 lz4 archive format, for which I do not have a full js decoder as of yet. I am unsure when I will as I have literally no time at the moment :/ The decode_stream.js just need to be adapted to browsers... If you find time, I accept pulls ;).

Cheers

[pierrec]

Ok... I have hacked together something that works in the browser (Chromium 31). It may help you although I consider this as unfinished work as I had to cut corners.

It available here: https://github.com/pierrec/node-lz4/blob/master/examples/www/uncompress.html

Only the decoder does work.

HTH

[tarelli]

@pierrec thanks for spending time on this! I see there is a new library lz4.js but the html file you pointed me to is the same old one. Do you have a short snippet for how to use the new library? Thanks! Btw, your help is very valuable to the OpenWorm project for which we are trying to use this library :)

[tarelli]

@pierrec I tried with var decompressedData = LZ4.decode(inputBuffer); and I get: Error: Invalid magic number: 1BBDBFEF @0

[tarelli]

An update. I switched to sending the data from the server in binary mode. The message being compressed is the following:

{"type":"read_url_parameters","data":"{}"}

The result of the compression is the following (both with the java porting and the JNI instance):

[-16, 27, 123, 34, 116, 121, 112, 101, 34, 58, 34, 114, 101, 97, 100, 95, 117, 114, 108, 95, 112, 97, 114, 97, 109, 101, 116, 101, 114, 115, 34, 44, 34, 100, 97, 116, 97, 34, 58, 34, 123, 125, 34, 125]

I am sending using ArrayBuffer. Client side when I receive the message I do the following:

var data= new Int8Array(msg.data);

Inspecting data gives:

data
[-16, 27, 123, 34, 116, 121, 112, 101, 34, 58, 34, 114, 101, 97, 100, 95, 117, 114, 108, 95, 112, 97, 114, 97, 109, 101, 116, 101, 114, 115, 34, 44, 34, 100, 97, 116, 97, 34, 58, 34, 123, 125, 34, 125]

which is the same thing that was sent, so far so good. If then I do:

var decompressedData = LZ4.decode(new Buffer(data));

I have the error Error:

Invalid magic number: 227B1BF0 @0

If I inspect new Buffer(data) I get:

[240, 27, 123, 34, 116, 121, 112, 101, 34, 58, 34, 114, 101, 97, 100, 95, 117, 114, 108, 95, 112, 97, 114, 97, 109, 101, 116, 101, 114, 115, 34, 44, 34, 100, 97, 116, 97, 34, 58, 34, 123, 125, 34, 125]

which is not the same as -16 became its two-complement 240. Is it a problem with Buffer? Am I doing something wrong? Thanks!

[pierrec]

Hello,

I have done some tests with your input data and I get totally different results from you: the LZ4 compress block function tells me it is not compressible... I tried in nodejs as the encoder doesnt work in the browser (yet).

If I encode using the LZ4 streaming format, I get: <Buffer 04 22 4d 18 64 70 b9 2b 00 00 80 7b 22 74 79 70 65 22 3a 22 72 65 61 64 5f 75 72 6c 5f 70 61 72 61 6d 65 74 65 72 73 22 2c 22 64 61 74 61 22 3a 22 7b 7d ...>

The uncompressed data is: <Buffer 7b 22 74 79 70 65 22 3a 22 72 65 61 64 5f 75 72 6c 5f 70 61 72 61 6d 65 74 65 72 73 22 2c 22 64 61 74 61 22 3a 22 7b 7d 22 7d 0a>

where the first 4 bytes are the magic number and where you see somewhere in there the uncompressed data (as it is not compressible).

I am using LZ4 r110. I wonder if the encoding phase does something to the data?

[pierrec]

Ok reposting as my Buffer data got mangled by github :s

uncompressed data:

<Buffer 7b 22 74 79 70 65 22 3a 22 72 65 61 64 5f 75 72 6c 5f 70 61 72 61 6d 65 74 65 72 73 22 2c 22 64 61 74 61 22 3a 22 7b 7d 22 7d 0a>

(not really) compressed data:

<Buffer 04 22 4d 18 64 70 b9 2b 00 00 80 7b 22 74 79 70 65 22 3a 22 72 65 61 64 5f 75 72 6c 5f 70 61 72 61 6d 65 74 65 72 73 22 2c 22 64 61 74 61 22 3a 22 7b 7d ...>

[mrx8]

Hi, I checked your code and it seems you only support streaming-format (with Magic-number and so on). However I need to decompress binary-data in node, received via dgram-packets, which is generated from a c-application using the code from https://code.google.com/p/lz4/. Could you also provide the plain LZ4_compress / LZ4_decompress from the C-library ?

[mrx8]

I just realized that you already provide "decodeBlock" and "encodeBlock" in lz4.js So all is fine :)