lz4 / lz4-java

LZ4 compression for Java
Apache License 2.0
1.09k stars 248 forks source link

Stream ended prematurely when trying to readAllBytes #210

Open yash29896 opened 1 year ago

yash29896 commented 1 year ago

So I'm trying to switch our compressor to lz4-java from snappy and I was trying something like this for a POC

When I try a readAllBytes() on the LZ4FrameInputStream it throw the below exception an breaks.

ERROR

Exception in thread "main" java.io.IOException: Stream ended prematurely
    at net.jpountz.lz4.LZ4FrameInputStream.readInt(LZ4FrameInputStream.java:245)
    at net.jpountz.lz4.LZ4FrameInputStream.readBlock(LZ4FrameInputStream.java:259)
    at net.jpountz.lz4.LZ4FrameInputStream.read(LZ4FrameInputStream.java:353)
    at java.base/java.io.InputStream.readNBytes(InputStream.java:396)
    at java.base/java.io.InputStream.readAllBytes(InputStream.java:333)
    at com.yash.lz4.App.main(App.java:22)

Calling Function

    public static void main( String[] args ) throws IOException
    {
        ByteHandler handler = new LZ4Compressor();
        byte[] b = {4, 34, 77, 24, 96, 112, 115, 6, 0, 0, -128, 3, 0, 1, 0, 0, 0};  // Test data from one of our unit tests
        InputStream inputStream = handler.deserialize(new ByteArrayInputStream(b));
        byte[] i = inputStream.readAllBytes();
    }

I've done some debugging and it seems like its not marking the frame as finished.

private void readBlock() throws IOException {
    // Reads Ok up to this point
    // Reads the last 6 bytes 3, 0, 1, 0, 0, 0  
    int blockSize = readInt(in);  
    final boolean compressed = (blockSize & LZ4FrameOutputStream.LZ4_FRAME_INCOMPRESSIBLE_MASK) == 0;
    blockSize &= ~LZ4FrameOutputStream.LZ4_FRAME_INCOMPRESSIBLE_MASK;  // blocksize = 6

    // Check for EndMark
    if (blockSize == 0) {           // Since this is 6 its doesnt enter the loop
......
      frameInfo.finish();
      return;
    }

Hence the next time it tries to read the stream which has already been completed it leaks into readBlock(); and breaks.

 public int read() throws IOException {
    while (!firstFrameHeaderRead || buffer.remaining() == 0) {
      if (!firstFrameHeaderRead || frameInfo.isFinished()) {
        if (firstFrameHeaderRead && readSingleFrame) {
          return -1;
        }
    if (!nextFrameInfo()) {
      return -1;
    }
      }
      readBlock();
    }

This is our Compressor implementation

public final class LZ4Compressor implements ByteHandler {

      public OutputStream serialize(final OutputStream toSerialize) {
        try {
          return new LZ4FrameOutputStream(toSerialize);
        } catch (IOException e) {
          throw new UncheckedIOException(e);
        }
      }

      public InputStream deserialize(final InputStream toDeserialize) {
        try {
          return new LZ4FrameInputStream(toDeserialize);
        } catch (IOException e) {
          throw new UncheckedIOException(e);
        }
      }

      public ByteHandler getInstance() {
        return new LZ4Compressor();
      }
    }