search

laudrup / lz4_stream

A C++ stream using LZ4 (de)compression
BSD 3-Clause "New" or "Revised" License
39 stars 13 forks source link

Input stream implementation is broken #2

Closed ilyail3 closed 5 years ago

ilyail3 commented 7 years ago

first of all, thanks for the help, this makes working with LZ4 more bearable.

Second, I tried using the input stream on the output of a large json filled file(I can't attach it since it was client's access logs) and discovered the output is getting truncated after about 14k.

I tracked the error down to the following line: if (dest_size == 0)

basically, lz4 doesn't HAVE to decompress a full block on this command, in some case the decompress function will return 0 decompressed because it didn't receive enough data to push anything out.

thus this if will EOF the stream prematurely.

What worked for me(extracted the whole file and the md5sum of the result was equal to lz4cat's) was to copy the example from LZ4's code https://github.com/lz4/lz4/blob/dev/examples/frameCompress.c

Here's the PR with my code: https://github.com/laudrup/lz4_stream/pull/1

I don't have much experience on C/CPP, so I probably made some mistakes, but this might help you find the issue

ilyail3 commented 7 years ago

I have to note that I also tried to comment out the dest_size==0 statement, the output size was correct but MD5sum wasn't, no clue why

laudrup commented 7 years ago

Hi there,

Thanks a lot for your kind words. I'm happy that someone can make use of this small project.

I'll look into fixing the issue you've experienced as soon as I get a bit more time. Then I'll get back to you.

Btw. your pull request doesn't build with clang and the unit tests fail when built with GCC.

I cannot promise when I'll have a look at this, but I hope soon.

laudrup commented 5 years ago

I believe this has been fixed in 77faf81b2e2bb1f22197274d8283ae0b99e3b1fe. Please let me know if that is not the case. Otherwise, I'll close this issue.

Thanks a lot.