Closed pbutler closed 9 years ago
In 4mc the LZ4 compression/decompression is bounded up to 4MB (4 * 1024 * 1024), which is an hard fixed limit. When creating a compressor or decompressor the direct buffer is allocated twice (once for compressed data buffer, and for output uncompressed data buffer). So each single decompressor istantiated will need no more than 2x 4MB direct buffers.
If you have some simple test case or directions to try and reproduce your issue, I can try.
This should be anyway fixed now by the direct buffer caching introduced, working good both on cluster distributed jobs and on HDFS clients using 4mc codecs.
When using this decompressor while decompressing the map outputs I run into problems with large files and it looks like the DirectBuffer grows too large. I have tried changing the maximum direct buffer size but it seems to require too large of a size to be practical (
-XX:MaxDirectMemorySize=2g
didn't work and I stopped there). The following exception is thrown:I am not sure if there is something that can be done on decompressor side to fix this or if I am just not setting the options correctly. Any help would be appreciated.