Closed mtopolnik closed 9 years ago
I agree with both points if they work good with following constraints:
If Hadoop needs many codec instances at once (e.g., one per core/thread) and if 4MC codecs use largish buffers, there may be a fundamental need for a lot of buffer space. However, the default direct memory limit appears to be equal to -Xmx
which seems like too much to cause OOME without an actual leak, or at least inefficient cleanup.
Out of interest, what is the magnitude of the impact on performance when no direct buffers are used? I'm thinking there may be customers who are prepared to sacrifice a small bit of perf for an improvement in stability.
4mc is designed for hadoop MR job cluster run, both hadoop 1.x and 2.x So e.g. in hadoop 1 you have 1 JVM for job but JVM can be reused; while in hadoop 2 with YARN in the end it's 1 JVM per job. Also, Hadoop is leveraging CodecPool to reuse compression codec instances during job exectuon. So all your ideas would be good to improve/enhance 4mc when it's used externally from hadoop, but within hadoop the impact will be almost negligible.
This is a continuation of the discussion which started on a LZ4 issue.
To summarize, I proposed:
byte[]
to natively accessible memory is inevitable either way;@carlomedas stated that the second approach still suffered from OOMEs. The main point with this approach is that there must be a clear demarcation of the direct buffer's lifecycle. Usually a job of some kind is started, causing the allocation of the direct buffer, and later the job is ended, providing the opportunity to clean up. If this aspect is missing and the only cleanup mechanism is the finalizer method, then this approach will be of no use. On the other hand, invoking
cleaner.clean()
will, to the best of my knowledge, unconditionally free the native memory block.