Make MAX_BLOCK_SIZE Configurable

[Maithem]

Hi,

What's the reason behind setting the MAX_BLOCK_SIZE to 32mb?

When used with netty, a common use case is to have a complete message within a single buffer. Based on what I understand, any buffer larger than the block size will be chunked. This limitation introduces complexity and adds overhead because the RPC-level encoder/decoder logic has to translate one message buffer to multiple compressed buffers and multiple compressed buffers to a single uncompressed message buffer.

I would like to compress buffers of size 50 - 100mb, is it reasonable to just extend the classes and bump up the MAX_BLOCK_SIZE constant?

Thanks, M

[odaira]

I assume you are talking about LZ4BlockOutputStream.MAX_BLOCK_SIZE. I was not involved in this project when that part was written, so I have no idea about the rationale behind the particular value of 32MB. It would be impossible to change the maximum block size without sacrificing the compatibility. If you don't have any concern about the compatibility (i.e. if you don't need to decompress the data that was compressed with a different maximum block size setting), I would fork the lz4-java project and increase LZ4BlockOutputStream.COMPRESSION_LEVEL_BASE, as you won't be able to change MAX_BLOCK_SIZE or COMPRESSION_LEVEL_BASE by simply extending the classes.

I have a question. Why do you have to consider the chunking inside the lz4 block stream? When you say "overhead", are you talking about coding overhead or performance overhead? Upper-layer applications like netty won't need to change even if chunking happens inside the lz4 block stream. In terms of the performance overhead, yes, the 32MB size would have more overhead than 50MB or 100MB, but did you see any visible difference in terms of the speed?

Another question is, why do you need to use the streaming interfaces (like LZ4BlockOutputStream)? If you have a buffer to compress in advance, why not using the lz4-java compressors and decompressors directly? They don't suffer from the chunking.

[Maithem]

I was mostly referring to coding overhead, I didn't benchmark performance with respect to block size.

To send a single 35mb rpc, the encoder will run on the first 32mb, then the remaining 3 (35 - 32=3), this will produce two buffers. Therefore we need to further encode the two buffers so that the decoder understands that those two buffers belong to the same RPC msg. This issue happens when using LengthFieldPrepender and LengthFieldBasedFrameDecoder.

Now if we can encode the whole 35mbs in one shot we don't need another layer of encoding to segment/combine messages that are larger than 32mb.

[Maithem]

I do have the full buffer in advance, now that I look at the code again it seems like the MAX_BLOCK_SIZE is only used in LZ4BlockOutputStream, are there similar size limitations for the lz4-java compressors (i.e. is it possible to encode arbitrary size buffers) ?

[odaira]

First, everything about the chunking is processed inside the lz4 block stream, so netty won't even be able to see what happens inside. From the netty (or any upper layer application) perspective, one stream (or buffer) is compressed into another stream (or buffer), not multiple streams (or buffers), even when chunking happens.

Second, the lz4-java compressor does not have any such a restriction. The only restriction I am aware of is the maximum size of a Java byte array, which I believe is 2GB.

[Maithem]

Makes sense. Thanks.