java.lang.OutOfMemoryError: Java heap space when creating an n-gram index

[GoogleCodeExporter]

Use the similar setting as the issue 922.

Use the generated twitter message dataset. 
The dataset size is 5GB. There are 5 partitions per machine,  6GB heap size,  
256MB in-memory components for each index.

Asterix Configuration
nc.java.opts                             :-Xmx6144m  -XX:MinHeapFreeRatio=5  
-XX:MaxHeapFreeRatio=15
cc.java.opts                             :-Xmx4096m -XX:MinHeapFreeRatio=5  
-XX:MaxHeapFreeRatio=15
max.wait.active.cluster                  :60
storage.buffercache.pagesize             :131072
storage.buffercache.size                 :1073741824
storage.buffercache.maxopenfiles         :214748364
storage.memorycomponent.pagesize         :131072
storage.memorycomponent.numpages         :2048
storage.metadata.memorycomponent.numpages:64
storage.memorycomponent.numcomponents    :2
storage.memorycomponent.globalbudget     :2147483648

When trying to create an n-gram index,
create index tNGramIdx on TweetMessages(message_text) type ngram(2);

the following exceptions are generated.

SEVERE: Uncaught Exception from thread Thread-14 message: Java heap space
java.lang.OutOfMemoryError: Java heap space
    at edu.uci.ics.hyracks.dataflow.std.sort.AbstractFrameSorter.sort(AbstractFrameSorter.java:111)
    at edu.uci.ics.hyracks.dataflow.std.sort.AbstractSortRunGenerator.flushFramesToRun(AbstractSortRunGenerator.java:55)
    at edu.uci.ics.hyracks.dataflow.std.sort.ExternalSortRunGenerator.nextFrame(ExternalSortRunGenerator.java:90)
    at edu.uci.ics.hyracks.dataflow.std.sort.AbstractSorterOperatorDescriptor$SortActivity$1.nextFrame(AbstractSorterOperatorDescriptor.java:133)
    at edu.uci.ics.hyracks.dataflow.common.comm.io.AbstractFrameAppender.flush(AbstractFrameAppender.java:79)
    at edu.uci.ics.hyracks.dataflow.common.comm.util.FrameUtils.appendToWriter(FrameUtils.java:158)
    at edu.uci.ics.hyracks.storage.am.lsm.invertedindex.dataflow.BinaryTokenizerOperatorNodePushable.nextFrame(BinaryTokenizerOperatorNodePushable.java:142)
    at edu.uci.ics.hyracks.dataflow.common.comm.io.AbstractFrameAppender.flush(AbstractFrameAppender.java:79)
    at edu.uci.ics.hyracks.dataflow.common.comm.util.FrameUtils.appendToWriter(FrameUtils.java:158)
    at edu.uci.ics.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.appendToFrameFromTupleBuilder(AbstractOneInputOneOutputOneFramePushRuntime.java:75)
    at edu.uci.ics.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.appendToFrameFromTupleBuilder(AbstractOneInputOneOutputOneFramePushRuntime.java:71)
    at edu.uci.ics.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.nextFrame(AssignRuntimeFactory.java:122)
    at edu.uci.ics.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$2.nextFrame(AlgebricksMetaOperatorDescriptor.java:136)
    at edu.uci.ics.hyracks.dataflow.common.comm.io.AbstractFrameAppender.flush(AbstractFrameAppender.java:79)
    at edu.uci.ics.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:192)
    at edu.uci.ics.hyracks.dataflow.std.misc.ConstantTupleSourceOperatorNodePushable.initialize(ConstantTupleSourceOperatorNodePushable.java:50)
    at edu.uci.ics.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:81)
    at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:251)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Original issue reported on code.google.com by wangs...@gmail.com on 12 Aug 2015 at 1:52

[GoogleCodeExporter]

This issue was found when checking the issue 922. I just checked the 
AbstractFrameSorter.sort part and confirmed that it was trying to allocate "new 
int[size]" where the size was 83800068. If we multiply this number by 4 bytes, 
then we gets 320MB. Jianfeng, it would be really nice if you could find out why 
it generates an OOM exception since his is related with the frame. :-)

Original comment by wangs...@gmail.com on 12 Aug 2015 at 1:56

[GoogleCodeExporter]

The int array is the pointer array storing the offset to the frame tuples which 
is created when the FrameMemoryBuffer is full. If the pointer is that big then 
the objects it pointed to could be even bigger.

The source of the problem is that the `frameLimit` parameter that sent to 
initialize the `ExternalSortRunGenerator` is too larger. 
Can you find out how big is that `frameLimit`? 
And what is the `compiler.framesize` and `compiler.sortmemory` in your settings?

Original comment by jianfeng...@gmail.com on 12 Aug 2015 at 3:02

[GoogleCodeExporter]

Thanks. Based on the sort memory, it makes sense? However, the problem is we 
have five partitions, and an OOM exception happens. 

compiler.framesize: 655,360 (650KB)
compiler.sortmemory: 335,544,320 (320MB)

I will try to check the frameLimit. 

Original comment by wangs...@gmail.com on 12 Aug 2015 at 3:10

[GoogleCodeExporter]

Yes, it makes sense if the `sortmemory` is 320MB. I guess your tuple is also 
very small, then the size of the pointer array is not neglectable. It will be a 
very tricky problems if we want to calculate the memory consumption very 
precisely, though it is a separate issue.

You can try to lower the `sortmemory` to 120MB or smaller to trigger the 2nd 
round ExternalSort.

Original comment by jianfeng...@gmail.com on 12 Aug 2015 at 3:24

[GoogleCodeExporter]

Yes. I will try that. 320MB for 6GB java heap was too much, maybe.

Original comment by wangs...@gmail.com on 12 Aug 2015 at 3:25

[GoogleCodeExporter]

While, the OOM exception is an overall effect. Each operator may hold several 
hundred of MB, sort operator is the last straw. 

Original comment by jianfeng...@gmail.com on 12 Aug 2015 at 4:40