search

fingltd / 4mc

4mc - splittable lz4 and zstd in hadoop/spark/flink
Other
108 stars 36 forks source link

Ideas to improve the handling of direct buffers #9

Closed mtopolnik closed 9 years ago

mtopolnik commented 9 years ago

This is a continuation of the discussion which started on a LZ4 issue.

To summarize, I proposed:

  1. dispensing with the direct buffer on the input side because the copying from byte[] to natively accessible memory is inevitable either way;
  2. using direct, but non-public, JDK API call to free the direct buffer's memory.

@carlomedas stated that the second approach still suffered from OOMEs. The main point with this approach is that there must be a clear demarcation of the direct buffer's lifecycle. Usually a job of some kind is started, causing the allocation of the direct buffer, and later the job is ended, providing the opportunity to clean up. If this aspect is missing and the only cleanup mechanism is the finalizer method, then this approach will be of no use. On the other hand, invoking cleaner.clean() will, to the best of my knowledge, unconditionally free the native memory block.

carlomedas commented 9 years ago

I agree with both points if they work good with following constraints:

  1. still works good with contracts of Hadoop compressors/decompressors
  2. if it works good from user code direct call and also automatically within the hadoop M/R jobs; and of course if it still avoids direct buffer OOM (which was not the case in my tests of couple months ago)
mtopolnik commented 9 years ago

If Hadoop needs many codec instances at once (e.g., one per core/thread) and if 4MC codecs use largish buffers, there may be a fundamental need for a lot of buffer space. However, the default direct memory limit appears to be equal to -Xmx which seems like too much to cause OOME without an actual leak, or at least inefficient cleanup.

Out of interest, what is the magnitude of the impact on performance when no direct buffers are used? I'm thinking there may be customers who are prepared to sacrifice a small bit of perf for an improvement in stability.

carlomedas commented 9 years ago

4mc is designed for hadoop MR job cluster run, both hadoop 1.x and 2.x So e.g. in hadoop 1 you have 1 JVM for job but JVM can be reused; while in hadoop 2 with YARN in the end it's 1 JVM per job. Also, Hadoop is leveraging CodecPool to reuse compression codec instances during job exectuon. So all your ideas would be good to improve/enhance 4mc when it's used externally from hadoop, but within hadoop the impact will be almost negligible.