airlift / aircompressor

A port of Snappy, LZO, LZ4, and Zstandard to Java
Apache License 2.0
562 stars 111 forks source link

Snappy Decompress throw MalformedInputException #148

Closed fatcarter closed 2 months ago

fatcarter commented 2 years ago

my code:

    private final static Compressor compressor = new SnappyCompressor();
    private final static Decompressor decompressor = new SnappyDecompressor();

    public byte[] compress(String s) {
        ByteBuffer byteBuffer = ByteBuffer.wrap(s.getBytes(StandardCharsets.UTF_8));
        int length = compressor.maxCompressedLength(s.getBytes(StandardCharsets.UTF_8).length);
        ByteBuffer out = ByteBuffer.allocate(length);
        compressor.compress(byteBuffer, out);
        return out.array();
    }

    public String decompress(byte[] s) {
        ByteBuffer in = ByteBuffer.wrap(s);
        int length = SnappyDecompressor.getUncompressedLength(s, 0);
        ByteBuffer out = ByteBuffer.allocate(length);
        decompressor.decompress(in, out);
        return new String(out.array());
    }

 public static void main(String[] args) {
        String str = "123";
        Logger log = LoggerFactory.getLogger("compress");
        log.info("char length={},byte array length={}", str.length(), str.getBytes(StandardCharsets.UTF_8).length);
        MessageCompresser compresser = new LzwMessageCompresser();
        byte[] compressedBytes = compresser.compress(str);
        String value = new String(compressedBytes);
        log.info("compressed:char length={},byte array length={}", value.length(), value.getBytes(StandardCharsets.UTF_8).length);
        log.info("value={}", value);
        String strs = compresser.decompress(compressedBytes);
        log.info("restore:{}", strs);
    }

why is MalformedInputException thrown when decompressing compressed bytes

fatcarter commented 2 years ago

when called main method:

13:39:00.620 [main] INFO compress - char length=3,byte array length=3
13:39:00.627 [main] INFO compress - compressed:char length=35,byte array length=35
13:39:00.627 [main] INFO compress - value=123
Exception in thread "main" io.airlift.compress.MalformedInputException: Malformed input: offset=5
    at io.airlift.compress.snappy.SnappyRawDecompressor.uncompressAll(SnappyRawDecompressor.java:124)
    at io.airlift.compress.snappy.SnappyRawDecompressor.decompress(SnappyRawDecompressor.java:53)
    at io.airlift.compress.snappy.SnappyDecompressor.decompress(SnappyDecompressor.java:101)
dain commented 2 months ago

I'm not sure what is wrong with the ByteBuffer code, but this will work:

    public byte[] compress(String s)
    {
        byte[] data = s.getBytes(StandardCharsets.UTF_8);
        int length = compressor.maxCompressedLength(data.length);
        byte[] out = new byte[length];
        int size = compressor.compress(data, 0, data.length, out, 0, out.length);
        return Arrays.copyOf(out, size);
    }

    public String decompress(byte[] compressed)
    {
        int length = decompressor.getUncompressedLength(compressed, 0);
        byte[] out = new byte[length];
        int size = decompressor.decompress(compressed, 0, compressed.length, out, 0, out.length);
        return new String(out,0, size, StandardCharsets.UTF_8);
    }

There are extensive tests for correctness that look like the above, so if you are looking for sample code checkout the tests.