airlift / aircompressor

A port of Snappy, LZO, LZ4, and Zstandard to Java
Apache License 2.0
549 stars 110 forks source link

ZstdOutputStream writes empty stream #169

Closed yarvik closed 1 month ago

yarvik commented 1 year ago

When I write bytes through ZstdOutputStream it pushes none of the compressed byes (or any bytes). Analogous code with Snappy works ok. Compressing bytes via new ZstdCompressor() then calling compress() method works ok.

Code example:

byte[] someTextBytes = new byte[]{90,115,116,100,79,117,116,112,117,116,83,116,114,101,97,109,32,
                    105,115,32,110,111,116,32,119,111,114,107,105,110,103,44,32,98,117,116,32,83,110,97,112,
                    112,121,70,114,97,109,101,100,79,117,116,112,117,116,83,116,114,101,97,109,32,105,115,
                    32,119,111,114,107,105,110,103,46,90,115,116,100,79,117,116,112,117,116,83,116,114,101,97,109,32,
                    105,115,32,110,111,116,32,119,111,114,107,105,110,103,44,32,98,117,116,32,83,110,97,112,
                    112,121,70,114,97,109,101,100,79,117,116,112,117,116,83,116,114,101,97,109,32,105,115,
                    32,119,111,114,107,105,110,103,46};

System.out.println("\n\nUnCompressed bytes to SnappyCompressOut -> System.out:");
System.out.println("----------------------------------------------------------");
SnappyFramedOutputStream snappyOS = new SnappyFramedOutputStream(System.out);
snappyOS.write(someTextBytes);
snappyOS.flush();
System.out.println("\n----------------------------------------------------------");
System.out.println("UnCompressed bytes to ZstdCompressOut -> System.out:");
System.out.println("----------------------------------------------------------");
ZstdOutputStream zstdOS = new ZstdOutputStream(System.out);
zstdOS.write(someTextBytes);
zstdOS.flush();
System.out.println("\n----------------------------------------------------------");

Result:

UnCompressed bytes to SnappyCompressOut -> System.out:
----------------------------------------------------------
)�.�I�I
----------------------------------------------------------
UnCompressed bytes to ZstdCompressOut -> System.out:
----------------------------------------------------------

----------------------------------------------------------

I tried file output stream, also tried wrapping various out streams in BufferedOutputStream, just to see where the issue lies. It looks like issue lies in private ZstdOutputStream.compressIfNecessary() where it decides not to .writeChunk()

yarvik commented 1 year ago

After playing with it some more, I found a way for it to work, by calling .close() on the stream. It appears that ZstdOutputStream does not implement/override flush() method (I am sure there's a reason for that), but .close() does write what's left in the buffer (via .writeChunk) So I just need to ensure I close the stream. So the issue is solved for me. Sorry for the alarm.

dain commented 1 month ago

The streams must be closed to function properly.