Cannot activate index since memory budget would be exceeded.

GoogleCodeExporter commented 8 years ago

What steps will reproduce the problem?

create asterix instance using managix one CC and NC.

step 1) Run these AQL statements from Web UI session one:

// DDL statements to drop, create and use a dataverse.
drop dataverse test if exists;
create dataverse test;
use dataverse test;

// DDL to create an open type
create type Cust as open {
cid: int32,
name: string
}

create dataset Customers(Cust) primary key cid;

create index idx-nm on Customers(name);

insert into dataset Customers({"cid":45,"name":"John Doe"});
insert into dataset Customers({"cid":35,"name":"Susan"});

// AQL Query - Indexed Nested Loops Join
for $c in dataset Customers
return $c 

step 2) Go to managix prompt and -
    (a) stop your asterix instance, next
    (b) start the stopped asterix instance

step 3) Run this select query from Web UI session two(different session):

use dataverse test;

for $c in dataset Customers
return $c 

The select query returns correct results.

Now, from Web UI session two, an attempt to insert data and then select on that 
dataset, gives this Exception

use dataverse test;

insert into dataset Customers({"cid":316,"name":"Tom"});
insert into dataset Customers({"cid":319,"name":"Bill"});

for $c in dataset Customers
return $c 

Cannot activate index since memory budget would be exceeded. 
[HyracksDataException]

Here is the stack trace from CC.log

SEVERE: Job failed on account of:
edu.uci.ics.hyracks.api.exceptions.HyracksDataException: Cannot activate index 
since memory budget would be exceeded.

edu.uci.ics.hyracks.api.exceptions.HyracksException: Job failed on account of:
edu.uci.ics.hyracks.api.exceptions.HyracksDataException: Cannot activate index 
since memory budget would be exceeded.

        at edu.uci.ics.hyracks.control.cc.job.JobRun.waitForCompletion(JobRun.java:190)
        at edu.uci.ics.hyracks.control.cc.work.WaitForJobCompletionWork$1.run(WaitForJobCompletionWork.java:44)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
Caused by: edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
edu.uci.ics.hyracks.api.exceptions.HyracksDataException: Cannot activate index 
since memory budget would be exceeded.
        at edu.uci.ics.hyracks.control.nc.Task.pushFrames(Task.java:320)
        at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:261)
        ... 3 more
Caused by: edu.uci.ics.hyracks.api.exceptions.HyracksDataException: Cannot 
activate index since memory budget would be exceeded.
        at edu.uci.ics.hyracks.storage.am.common.dataflow.IndexLifecycleManager.open(IndexLifecycleManager.java:91)
        at edu.uci.ics.hyracks.storage.am.common.dataflow.IndexDataflowHelper.open(IndexDataflowHelper.java:109)
        at edu.uci.ics.hyracks.storage.am.common.dataflow.IndexInsertUpdateDeleteOperatorNodePushable.open(IndexInsertUpdateDeleteOperatorNodePushable.java:67)
        at edu.uci.ics.hyracks.control.nc.Task.pushFrames(Task.java:296)
        ... 4 more

The Exception is coming from
/hyracks/hyracks-storage-am-common/src/main/java/edu/uci/ics/hyracks/storage/am/
common/dataflow/IndexLifecycleManager.java

Configuration details from managix

kfmohamm@khurram:~/asterix-managix/bin$ ./managix describe -admin
INFO: Name:astx01
Created:Tue Jun 04 08:58:15 PDT 2013
Web-Url:http://127.0.0.1:19001
State:ACTIVE (Tue Jun 04 08:59:19 PDT 2013)
Master node:master:127.0.0.1
node1:127.0.0.1

Asterix version:0.0.6-SNAPSHOT
Metadata Node:node1
Processes
NC at node1 [ 1988 ]
CC at master [ 1654 ]

Asterix Configuration
nc.java.opts:-Xmx1024m
cc.java.opts:-Xmx1024m
storage.buffercache.pagesize:32768
storage.buffercache.numpages:1024
storage.buffercache.maxopenfiles:214748364
storage.memorycomponent.pagesize:32768
storage.memorycomponent.numpages:4096
storage.memorycomponent.globalbudget:263435456
storage.lsm.mergethreshold:3
storage.lsm.bloomfilter.falsepositiverate:0.01
txn.log.buffer.numpages:8
txn.log.buffer.pagesize:131072
txn.log.partitionsize:2147483648
txn.log.groupcommitinterval:200
txn.log.checkpoint.lsnthreshold:67108864
txn.log.checkpoint.pollfrequency:120
txn.lock.escalationthreshold:1000
txn.lock.shrinktimer:120000
compiler.sortmemory:536870912
compiler.joinmemory:536870912
compiler.framesize:32768
web.port:19001
api.port:19101
log.level:INFO

Details of asterix code base revision: (this was the latest change available in 
my build)

kfmohamm@khurram:~/asterixdb_release/asterixdb$ git log
commit 5a05d466c779b4a87e5e21c20af369b42777cf79
Author: Madhusudan.C.S <madhusudancs@gmail.com>
Date:   Tue Jun 4 02:14:52 2013 -0700

    Show the error messages for parser exception similar to the other exceptions.

Sattam assigning this to you, please feel free to re-assign to owner, if you do 
not own it.

Original issue reported on code.google.com by khfaraaz82 on 4 Jun 2013 at 4:11

GoogleCodeExporter commented 8 years ago

This check at line 89, causes the Exception to be thrown, from the 
hyracks/storage/am/common/dataflow/IndexLifecycleManager.java source file.

 while (memoryUsed + inMemorySize > memoryBudget) {
            if (!evictCandidateIndex()) {
                throw new HyracksDataException("Cannot activate index since memory budget would be exceeded.");
            }
        }

Original comment by khfaraaz82 on 4 Jun 2013 at 4:15

GoogleCodeExporter commented 8 years ago

Sattam, Zach & I were able to reproduce the same Exception on my machine now. 
Interestingly, this time I did not have to go past step one mentioned above. 
Executing these (see below) steps, will lead to the Exception.

NOTE: The Exception is thrown when I do the second insert, up until the first 
insert statement, there is no Exception, we see it when we do the second insert.

// DDL statements to drop, create and use a dataverse.
drop dataverse test if exists;
create dataverse test;
use dataverse test;

// DDL to create an open type
create type Cust as open {
cid: int32,
name: string
}

create dataset Customers(Cust) primary key cid;

create index idx-nm on Customers(name);

insert into dataset Customers({"cid":45,"name":"John Doe"});
insert into dataset Customers({"cid":35,"name":"Susan"});

Cannot activate index since memory budget would be exceeded. 
[HyracksDataException]

Original comment by khfaraaz82 on 4 Jun 2013 at 7:32

GoogleCodeExporter commented 8 years ago

Marking it as invalid.

Original comment by salsuba...@gmail.com on 4 Jun 2013 at 9:13

Changed state: Invalid

GoogleCodeExporter commented 8 years ago

Why is this marked as invalid ?

Original comment by khfaraaz82 on 4 Jun 2013 at 9:33

GoogleCodeExporter commented 8 years ago

There is no issue with restarting managix. The issue is the configured
parameter for the memory budget is low.

Original comment by salsuba...@gmail.com on 4 Jun 2013 at 10:02

GoogleCodeExporter commented 8 years ago

Thanks Sattam, the issue does not show up after increasing the parameter 
storage.memorycomponent.globalbudget value to 1073741824 (i.e., 1GB)

Original comment by khfaraaz82 on 4 Jun 2013 at 10:22

GoogleCodeExporter commented 8 years ago

Re-opening this issue since I heard from Zack that he saw weird behavior. Will 
investigate.

Original comment by salsuba...@gmail.com on 4 Jun 2013 at 10:53

Changed state: New

GoogleCodeExporter commented 8 years ago

Vinayak, Sattam and Zach.

As discussed in our 5:30 PM meeting, Sattam & I tried to reproduce issue 499 
reported here, on Zach's branch.

However, I hit this other ArrayIndexOutOfBoundsException on Zach's branch.

Here is the stack trace.

edu.uci.ics.asterix.metadata.MetadataException: 
java.lang.ArrayIndexOutOfBoundsException: 2
    at edu.uci.ics.asterix.metadata.MetadataNode.addDataset(MetadataNode.java:184)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
    at sun.rmi.transport.Transport$1.run(Transport.java:177)
    at sun.rmi.transport.Transport$1.run(Transport.java:174)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
    at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:273)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:251)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:160)
    at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:194)
    at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:148)
    at com.sun.proxy.$Proxy13.addDataset(Unknown Source)
    at edu.uci.ics.asterix.metadata.MetadataManager.addDataset(MetadataManager.java:225)
    at edu.uci.ics.asterix.aql.translator.AqlTranslator.handleCreateDatasetStatement(AqlTranslator.java:455)
    at edu.uci.ics.asterix.aql.translator.AqlTranslator.compileAndExecute(AqlTranslator.java:203)
    at edu.uci.ics.asterix.api.http.servlet.APIServlet.doPost(APIServlet.java:85)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:546)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:483)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:970)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:411)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:904)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110)
    at org.eclipse.jetty.server.Server.handle(Server.java:347)
    at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:439)
    at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:924)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:781)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:220)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:43)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:545)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:43)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
    at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
    at edu.uci.ics.hyracks.storage.am.common.tuples.TypeAwareTupleWriter.getFieldSlotsBytes(TypeAwareTupleWriter.java:122)
    at edu.uci.ics.hyracks.storage.am.common.tuples.TypeAwareTupleWriter.bytesRequired(TypeAwareTupleWriter.java:36)
    at edu.uci.ics.hyracks.storage.am.lsm.btree.tuples.LSMBTreeTupleWriter.bytesRequired(LSMBTreeTupleWriter.java:39)
    at edu.uci.ics.hyracks.storage.am.btree.frames.BTreeNSMLeafFrame.getBytesRequriedToWriteTuple(BTreeNSMLeafFrame.java:50)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree.upsert(BTree.java:324)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree.access$500(BTree.java:68)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree$BTreeAccessor.upsertIfConditionElseInsert(BTree.java:895)
    at edu.uci.ics.hyracks.storage.am.lsm.btree.impls.LSMBTree.insert(LSMBTree.java:306)
    at edu.uci.ics.hyracks.storage.am.lsm.btree.impls.LSMBTree.modify(LSMBTree.java:258)
    at edu.uci.ics.hyracks.storage.am.lsm.common.impls.LSMHarness.modify(LSMHarness.java:131)
    at edu.uci.ics.hyracks.storage.am.lsm.common.impls.LSMHarness.modify(LSMHarness.java:122)
    at edu.uci.ics.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.insert(LSMTreeIndexAccessor.java:42)
    at edu.uci.ics.asterix.metadata.MetadataNode.insertTupleIntoIndex(MetadataNode.java:280)
    at edu.uci.ics.asterix.metadata.MetadataNode.addDataset(MetadataNode.java:174)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
    at sun.rmi.transport.Transport$1.run(Transport.java:177)
    at sun.rmi.transport.Transport$1.run(Transport.java:174)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    ... 1 more

Original comment by khfaraaz82 on 5 Jun 2013 at 1:55

GoogleCodeExporter commented 8 years ago

Here is the value of the parameter that I used to reproduce the issue

 <property>
                <name>storage.memorycomponent.globalbudget</name>
                <value>256120192</value>

Original comment by khfaraaz82 on 5 Jun 2013 at 1:59

GoogleCodeExporter commented 8 years ago

Original comment by vinay...@gmail.com on 5 Jun 2013 at 6:25

Added labels: Severity-BetaBlocker

GoogleCodeExporter commented 8 years ago

Original comment by vinay...@gmail.com on 5 Jun 2013 at 7:06

Added labels: Severity2-RC1Blocker

GoogleCodeExporter commented 8 years ago

LGTM.

Original comment by buyingyi@gmail.com on 5 Jun 2013 at 8:08

GoogleCodeExporter commented 8 years ago

Original comment by vinay...@gmail.com on 5 Jun 2013 at 8:12

Changed state: Fixed

GoogleCodeExporter commented 8 years ago

Original comment by khfaraaz82 on 5 Jun 2013 at 11:05

Changed state: Verified

br1ghtyang / asterixdb

Cannot activate index since memory budget would be exceeded. #499