Closed sahilpaudel-pe closed 3 years ago
Thanks for reporting. I think there are two issue here.
Decompressing data that was compressed by go or ruby clients: LZ4DecompressorWithLength is not meant for this purpose. It is used for decompressing data compressed by LZ4CompressorWithLength, as stated in the document. Depending on how you compressed data using go or ruby, you may want to use either LZ4*Decompressor or LZ4InputStream. If you could give me a sample of compressed data or sample go/ruby code for compression, I can figure it out.
SIGBUS error: No matter how you use the lz4-java API, a SIGBUS error should not happen. Could you give me a sample compressed data that caused this error? Also, could you tell me the following information?
Sample Compressed Data (Go):
8xV7CiAgICJzdGF0dXMiOjEsCiAgICJkYXRhIjpbCiAgICAgIHsIAPcZICAgIm5hbWUiOiJhdXRvbWF0aW9uXzExMDQ0NzIyMTEzNzI2OTgiLC8A+RBkZXNjcmlwdGlvbiI6ImNyZWF0ZSBleHBlcmltZW50LAD5DGdvYWwiOiJUZXN0IHdpdGggdmFsaWQgZGF0YSgARnRhZ3OdAAICAKgiaG9tZV9wYWdlfwCSICAgImxvZ2luGgAGRAAYXVAArWlkZW50aWZpZXJXAAEdAQgCAMcic2VnbWVudCI6OTVEAAICAP4BInBsYXRmb3JtIjoiaW9zIiEA/gRkZXZpY2VfdmVyc2lvbiI6NS42JQBiZXhwcmVzIQAEcABZPT05NSIqABZ9DgAJzgBKbWV0Yb8BAgIA/AMiQVVUT01BVElPTl9URVNUXzEqAQwhAB8yIQANFzOSAAmEAAXgAA9OAQ3JQ3JlYXRlZEJ5Ijp7KwACAgAiIk5uAmx0ZXN0IHDbAQICAPsNIkVtYWlsIjoidGVzdHBlQHBoYXJtZWFzeS5pbsEAAgIA9xYiRXZlbnRUaW1lIjoiMjAyMC0wNS0yMVQwMjo1MjoyMy44NTZa2gACAgAdfUsAvyJTd2l0Y2hlZE9uwAB8LzkxwAASn1R1cm5lZE9mZr8ADQ9HAQQDeAEPHgAEB2UB/wQwMDAxLTAxLTAxVDAwOjAwOjAwYQESb0NvbXBsZSACEA+DAAQPoQBOb0FyY2hpdsACDw+gAGXdZXhwb3N1cmUiOjEwMIQAjyJzdGFydF90CgMGHlq4AP8HInNjaGVkdWxlcl9jaGVjayI6dHJ1ZVsAAo90dXMiOiJvbkUAAP8Ad2lubmluZ192YXJpYW50IgECB5cFDEgEAgIADZwFAgIABKIGCDkFDYIAAgIAfyJzcGxpdF8uBQEIAgAPaQAEAgIAfyJpZCI6IkFjAAUCAgACaQBPIjo0MCcAC89yZXR1cm5fdmFsdWVSBAMIAgAP0gUATjoiNTBwAQsCAA04AF8yIjoiMzgAIG0zIjoiMjDKAwgCAAemBgsCAA5RBQUCAA9xARAfQq4ACwRxAR429gIIAgAPcQExDzkBIV8yIjoiM5cCBgUCAA2pARIzOAAPHwEJD3EBCA8aAAMfXRcAAA1FAAolAAcfAAAZANAgICAgfQogICBdCn0K
Lz4-java: 1.7.1 OS: MacOS (10.15.6) Processor: 1.4 GHz Quad-Core Intel Core i5
Thanks for the sample data. It's in the LZ4 block format. Please use LZ4FastDecompressor or LZ4SafeDecompressor, not LZ4DecompressorWithLength, to decompress it.
I'll work on how to handle the runtime error....
I have the same question about how you decompress data without access to the size of the decompressed data -- the docs mention that the buffer should be "oversize" but doesn't specify by how much.
@wsargent
I got it working with below code
LZ4Factory safeFactory = LZ4Factory.safeInstance();
LZ4SafeDecompressor decompressor = safeFactory.safeDecompressor();
byte[] decoded = Base64.getDecoder().decode(data);
byte[] decomp = new byte[decoded.length * 5];
decompressor.decompress(decoded, decomp);
System.out.println(new String(decomp));
Hope this helps.
Although sometimes I get `net.jpountz.lz4.LZ4Exception: Error decoding offset 733 of input buffer` exception and all time when using LZ4FastDecompressor.
@sahilpaudel-pe according to https://github.com/lz4/lz4-java/pull/168#issuecomment-705940814 it can be up to 256x the compressed size so it looks like using the frame oriented outputstream/inputstream is the way to go, although I don't know how you'd know at first glance whether the other client was using a block or frame format -- presumably you look at the header for magic numbers...
https://github.com/lz4/lz4/blob/master/doc/lz4_Frame_format.md
@sahilpaudel-pe It is not possible to completely avoid SIGBUS/SIGSEGV when using LZ4FastDecompressor with unknown input. By e4780ab9789fe3a8e4a144d1d26e2815311ed502, I have supported using LZ4SafeDecompressor in LZ4DecompressorWithLength, so that a Java exception will be thrown instead of SIGBUS or SIGSEGV when it is used with a malformed input. Thanks for the heads-up!
I have a requirement where some data that may be encoded using go lz4 clients or ruby lz4 clients. How can I decode that using lz4-java.
I tried using
But I got strange error
Normal java to java decoding is processed as expected.
Thanks.