Open tomerz90 opened 4 years ago
The LZ4 block format does not use a checksum, but the LZ4 frame format does use a checksum. https://github.com/lz4/lz4/blob/master/doc/lz4_Frame_format.md I don't think it is weird for a streaming compression format to use a checksum.
Although LZ4BlockOutputStream forces you to use a checksum, LZ4FrameOutputStream allows you to disable it by not specifying CONTENT_CHECKSUM
or BLOCK_CHECKSUM
.
I never said its weird :), Im was just trying to understand its purpose.
I don't remember I have seen any document regarding LZ4Frame vs LZ4Block. I have never measured them myself for comparison, but the 13% difference is not something that can be ignored. Actually, I happen to be writing a benchmark suite to compare Java streaming compression algorithms, so I am going to reproduce your observation soon. Hopefully, I would like to close the 13% gap, rather than maintaining LZ4Block* forever....
AFAIK, its not some "convention" or something for output streams and also lz4 doesnt use a checksum. From my tests it uses significant CPU and if its there for the reason I think it is, it is unnecessary for me and I can use a "NO-OP" checksum. Can you please explain why LZ4BlockOutputStream needs a checksum?