pierrec / node-lz4

LZ4 fast compression algorithm for NodeJS
MIT License
438 stars 98 forks source link

Invalid magic number #11

Closed miqmago closed 10 years ago

miqmago commented 10 years ago

Hi, I'm using lz4 c library to compress a string in a c++ project, send it to a nodejs server in b64 and decompress there. To compress the string (buffer is comming from rapidjson: GetString() is a well formed json char * and Size() is the correct size):

  int safeSize = LZ4_compressBound(buffer.Size());
  char *compressed = new char[safeSize];
  int compressedBytes = LZ4_compress(buffer.GetString(), compressed, buffer.Size());
  string ret(BASE64::base64_encode((unsigned char *) compressed, compressedBytes));
  delete [] compressed;

The b64 string received in server perfectly matches the sended b64 string. To decompress the string in nodejs:

  var compressedData = new Buffer(req.query.d, 'base64');
  var decompressedData = LZ4.decode(compressedData);

I'm always getting Error: Invalid magic number: 7B5B74F1

I've been investigating c library and there is nothing similar to a magic number. Is there something to do with versions? I've seen that lz4.cli.c (used in node.js) has LZ4S_MAGICNUMBER defined and some other differences to lz4.c. In that case, would it work if I compile the library with c files from node.js to use it in c++? OTW how should I proceed?

Please any help is really appreciated.

Regards, Miguel

miqmago commented 10 years ago

Ok, I've seen, correct me if I'm wrong, that this is to decode files.

If it can help anyone, I've managed to decode strings:

var LZ4Lib = require('lz4/build/Release/lz4');
exports.LZ4_uncompress = function(input, outputSize) {
  var output;
  if (typeof outputSize !== 'undefined') {
    output = new Buffer(outputSize);
    LZ4Lib.uncompress(input, output);
  } else {
    outputSize = input.length * 2; // TODO: calculate a safe outputSize form input size
    output = new Buffer(outputSize);
    LZ4Lib.uncompress_unknownOutputSize(input, output);
  }
  return output;
};
pierrec commented 10 years ago

Hello,

Indeed the exposed function decode() is to decode a file. What you are trying to do is to decode a chunk of compressed data. The function does exist but it is not exposed. I may do so in the future though.

For now, you could do something like what you suggested:

var uncompress = require('lz4').createDecoderStream.prototype.uncompressBlock

where uncompress() expects an input and output buffer.

HTH