So I'm trying to switch our compressor to lz4-java from snappy and I was trying something like this for a POC
When I try a readAllBytes() on the LZ4FrameInputStream it throw the below exception an breaks.
ERROR
Exception in thread "main" java.io.IOException: Stream ended prematurely
at net.jpountz.lz4.LZ4FrameInputStream.readInt(LZ4FrameInputStream.java:245)
at net.jpountz.lz4.LZ4FrameInputStream.readBlock(LZ4FrameInputStream.java:259)
at net.jpountz.lz4.LZ4FrameInputStream.read(LZ4FrameInputStream.java:353)
at java.base/java.io.InputStream.readNBytes(InputStream.java:396)
at java.base/java.io.InputStream.readAllBytes(InputStream.java:333)
at com.yash.lz4.App.main(App.java:22)
Calling Function
public static void main( String[] args ) throws IOException
{
ByteHandler handler = new LZ4Compressor();
byte[] b = {4, 34, 77, 24, 96, 112, 115, 6, 0, 0, -128, 3, 0, 1, 0, 0, 0}; // Test data from one of our unit tests
InputStream inputStream = handler.deserialize(new ByteArrayInputStream(b));
byte[] i = inputStream.readAllBytes();
}
I've done some debugging and it seems like its not marking the frame as finished.
private void readBlock() throws IOException {
// Reads Ok up to this point
// Reads the last 6 bytes 3, 0, 1, 0, 0, 0
int blockSize = readInt(in);
final boolean compressed = (blockSize & LZ4FrameOutputStream.LZ4_FRAME_INCOMPRESSIBLE_MASK) == 0;
blockSize &= ~LZ4FrameOutputStream.LZ4_FRAME_INCOMPRESSIBLE_MASK; // blocksize = 6
// Check for EndMark
if (blockSize == 0) { // Since this is 6 its doesnt enter the loop
......
frameInfo.finish();
return;
}
Hence the next time it tries to read the stream which has already been completed it leaks into readBlock(); and breaks.
public int read() throws IOException {
while (!firstFrameHeaderRead || buffer.remaining() == 0) {
if (!firstFrameHeaderRead || frameInfo.isFinished()) {
if (firstFrameHeaderRead && readSingleFrame) {
return -1;
}
if (!nextFrameInfo()) {
return -1;
}
}
readBlock();
}
This is our Compressor implementation
public final class LZ4Compressor implements ByteHandler {
public OutputStream serialize(final OutputStream toSerialize) {
try {
return new LZ4FrameOutputStream(toSerialize);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
public InputStream deserialize(final InputStream toDeserialize) {
try {
return new LZ4FrameInputStream(toDeserialize);
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
public ByteHandler getInstance() {
return new LZ4Compressor();
}
}
So I'm trying to switch our compressor to lz4-java from snappy and I was trying something like this for a POC
When I try a readAllBytes() on the LZ4FrameInputStream it throw the below exception an breaks.
ERROR
Calling Function
I've done some debugging and it seems like its not marking the frame as finished.
Hence the next time it tries to read the stream which has already been completed it leaks into readBlock(); and breaks.
This is our Compressor implementation