luben / zstd-jni

JNI binding for Zstd
Other
809 stars 165 forks source link

Having trouble decompressing a file which was compressed using --long=31 option with a ZstdDirectBufferDecompressingStream #231

Closed VladDm93 closed 1 year ago

VladDm93 commented 1 year ago
  1. Compress a file larger than 128mb with an option --long=31
PS D:\test2> .\zstd.exe --long=31 .\hello.txt
.\hello.txt          :  0.01%   (   341 MiB =>   33.6 KiB, .\hello.txt.zst)
  1. try to decompress the file without --long=31

    PS D:\test2> .\zstd.exe -d .\hello.txt.zst
    zstd: .\hello.txt already exists; overwrite (y/n) ? y
    .\hello.txt.zst : Decoding error (36) : Frame requires too much memory for decoding
    .\hello.txt.zst : Window size larger than maximum : 357166908 > 134217728
    .\hello.txt.zst : Use --long=29 or --memory=341MB
  2. try to decompress with --long=31 option. Success!

    PS D:\test2> .\zstd.exe -d --long=31 .\hello.txt.zst
    .\hello.txt.zst     : 357166908 bytes

While trying to decompress the same file using ZstdDirectBufferDecompressingStream I'm getting the same error and I'm failing to find a way to pass a --long or a --memory option

java.io.IOException: Frame requires too much memory for decoding
    at com.github.luben.zstd.ZstdDirectBufferDecompressingStreamNoFinalizer.read(ZstdDirectBufferDecompressingStreamNoFinalizer.java:89)
    at com.github.luben.zstd.ZstdDirectBufferDecompressingStream.read(ZstdDirectBufferDecompressingStream.java:66)
File f = new File("hello.txt.zst");
FileChannel fileChannel = FileChannel.open(f.toPath(), StandardOpenOption.READ);
MappedByteBuffer fileBuffer = fileChannel.map(FileChannel.MapMode.READ_ONLY, 0, fileChannel.size());

int size = 1024 * 50;
ByteBuffer streamBuffer = ByteBuffer.allocateDirect(size);
while (streamBuffer.hasRemaining() && fileBuffer.hasRemaining()) {
    streamBuffer.put(fileBuffer.get());
}
streamBuffer.flip();

ZstdDirectBufferDecompressingStream zis = new ZstdDirectBufferDecompressingStream(streamBuffer) {
    @Override
    protected ByteBuffer refill(ByteBuffer toRefill) {
        if (fileBuffer.hasRemaining()) {
            toRefill.clear();
            while (toRefill.hasRemaining() && fileBuffer.hasRemaining()) {
                toRefill.put(fileBuffer.get());
            }
            toRefill.flip();
        }
        return toRefill;
    }
};
ByteBuffer block = ByteBuffer.allocateDirect(1024 * 100);

while (zis.hasRemaining()) {
    block.clear();
    zis.read(block);
    block.flip();

    byte[] out = new byte[block.remaining()];
    block.get(out);
    System.out.println(new String(out, StandardCharsets.UTF_8));
}

I have tried to use a Zstd.setCompressionLong (and getting stream pointer from com.github.luben.zstd.ZstdDirectBufferDecompressingStreamNoFinalizer#stream using reflection) but i was keep getting the same error.

luben commented 1 year ago

I have just released v1.5.2-4 that adds setLongMax method to allow decompressing payloads that were compressed with non-default long mode.