Run 1GB MapReduce on YARN

[johnyangk]

Need to fix parallelism, data read, and OOM bugs.

[johnyangk]

1GB=9HDFS blocks, but we get a single Reader: Beam bug? The single reader takes 3 minutes to read its data(maybe it is reading 1GB?) The map operation that follows the read hogs cpu/memory(4 cores, 20GB).

[johnyangk]

I've skimmed through the Beam-HDFS adapter that we're using(v.0.6), and the code seems to be not using the HDFS APIs correctly.

In the latest Beam version(v.2.0) the corresponding code has completely changed for the better, and looks 'right' to me.

Thus, I conclude that this issue is blocked by #210.

[johnyangk]

Related Beam Issue: https://github.com/apache/beam/pull/2908

[johnyangk]

I/O from external sources seems to be transparently supported from Beam 2.0 😄
https://beam.apache.org/documentation/programming-guide/#io

[johnyangk]

Maybe not. Still not sure how to write to HDFS with v2.0 without the removed HDFSFileSink.

[bgchun]

Can you send an email to the beam mailing list?

Sent from my iPhone

2017. 6. 2. 오전 10:50 John Yang notifications@github.com 작성:

Maybe not. Still not sure how to write to HDFS with v2.0 without the removed HDFSFileSink.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[johnyangk]

@bgchun Sure, will do.

[johnyangk]

There has been some progress

• Beam 2.0.0 => Parallelism is now set correctly(9 tasks, each with 128MB), read time is reasonable (the previous version was buggy after all)
• Remove data from hashmap on LocalBlockStore#getBlock => Fixes memory leak that incurs GC hanging with just 1 map task

However, the executor still experiences GC hanging with around 5th map task. Probably there are more memory leaks that need to be fixed...

[bgchun]

Java memory profiling https://dzone.com/articles/java-memory-model-simplified

[johnyangk]

Thanks @bgchun. I guess the best way to find out would be to monitor the executor JVM. 😄 Note that the JVM memory size of the executor that I'm experimenting with is 20GB and the data size is 1GB.

[johnyangk]

Intermediate data still seems to be referenced from somewhere... Some stats from jmap:

ubuntu@optiplex-7040-17:~$ sudo jmap -heap 17174
Attaching to process ID 17174, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.101-b13

using thread-local object allocation.
Parallel GC with 4 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 0
   MaxHeapFreeRatio         = 100
   MaxHeapSize              = 18897436672 (18022.0MB)
   NewSize                  = 175112192 (167.0MB)
   MaxNewSize               = 6298796032 (6007.0MB)
   OldSize                  = 351272960 (335.0MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 2100297728 (2003.0MB)
   used     = 2100297728 (2003.0MB)
   free     = 0 (0.0MB)
   100.0% used
From Space:
   capacity = 2099249152 (2002.0MB)
   used     = 0 (0.0MB)
   free     = 2099249152 (2002.0MB)
   0.0% used
To Space:
   capacity = 2099249152 (2002.0MB)
   used     = 0 (0.0MB)
   free     = 2099249152 (2002.0MB)
   0.0% used
PS Old Generation
   capacity = 12598640640 (12015.0MB)
   used     = 12598584024 (12014.946006774902MB)
   free     = 56616 (0.05399322509765625MB)
   99.99955061818478% used

9396 interned Strings occupying 845384 bytes.
ubuntu@optiplex-7040-17:~$ jmap -histo:live 17174 | head

 num     #instances         #bytes  class name
----------------------------------------------
   1:     124373904     3127925384  [B
   2:     124372522     2984940528  org.apache.hadoop.io.Text
   3:     124372492     2984939808  org.apache.hadoop.io.LongWritable
   4:     124372491     2984939784  org.apache.beam.sdk.values.KV
   5:     124372492     1989959872  edu.snu.vortex.compiler.frontend.beam.BeamElement
   6:          3910      608511640  [Ljava.lang.Object;
   7:         21703        2011064  [C

[johnyangk]

It seems that the OOM is caused not by memory leak, but by Java object overhead with the synthetic dataset.

The 1gb synthetic dataset(sample_input_mr_1gb) that I've been using is consisted of a large number of very small tuples(e.g., "john 1").

As JVM wraps each tuple with nested objects(BeamElement-KV-String(Char array)/Long) for processing, the dataset ends up creating a sea of tiny objects in the executor JVM.

The overhead is shown in the jmap result below. For example, BeamElement is just a wrapper object and does not have fields other than KV, but it takes up a whopping 1.7GB of memory. However, the number makes sense if you divide it with the number of object instances to get 16 bytes per object, which is exactly the minimum Java object size.

ubuntu@optiplex-7040-16:~$ jmap -histo:live 30581 | head

 num     #instances         #bytes  class name
----------------------------------------------
   1:     114662012     3801523352  [C
   2:     114661734     2751881616  java.lang.String
   3:     114640684     2751376416  java.lang.Long
   4:     114640371     2751368904  org.apache.beam.sdk.values.KV
   5:     114640372     1834245952  edu.snu.vortex.compiler.frontend.beam.BeamElement
   6:          3943      560201480  [Ljava.lang.Object;
   7:           273        1085856  [Ljava.nio.channels.SelectionKey;

We did not see this problem with Pado, because the Wikipedia pageview dataset we used had much larger data per row as the following example:

aa %CE%92%CE%84_%CE%95%CF%80%CE%B9%CF%83%CF%84%CE%BF%CE%BB%CE%AE_%CE%99%CF%89%CE%AC%CE%BD%CE%BD%CE%B7/el/%CE%92 1 4854

Tomorrow, I'll write a new MapReduce application for the Wikipedia dataset, and see if I can run 1GB with it. 😄