Null pointer exception in DatasetLifecycleManager

[GoogleCodeExporter]

Reported by Christopher McIntyre and Till:

If  q02 is run before q03 or q03 is runbefore q02 in the web interface, the 
user gets the message "null[]".  This appears to indicate that a null pointer 
exception was thrown.  If the managix stop command is run after this, the stop 
command never finishes or at least takes an abnormally long amount of time.  At 
this point, some asterix processes must be killed on each of the nodes in the 
cluster, "managix describe" must be run twice (this causes the state to go from 
"ACTIVE" to "UNUSABLE" to "INACTIVE"), and then the managix start command can 
be run to restart the asterix instance.  The following is the stack trace for 
the exception thrown when q02 is run before q03 or vice versa is ran:

SEVERE: edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
java.lang.NullPointerException
edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
java.lang.NullPointerException
    at edu.uci.ics.hyracks.client.dataset.HyracksDatasetReader.read(HyracksDatasetReader.java:110)
    at edu.uci.ics.asterix.result.ResultReader.read(ResultReader.java:57)
    at edu.uci.ics.asterix.aql.translator.AqlTranslator.handleQuery(AqlTranslator.java:1416)
    at edu.uci.ics.asterix.aql.translator.AqlTranslator.compileAndExecute(AqlTranslator.java:279)
    at edu.uci.ics.asterix.api.http.servlet.APIServlet.doPost(APIServlet.java:96)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:546)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:483)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:970)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:411)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:904)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110)
    at org.eclipse.jetty.server.Server.handle(Server.java:347)
    at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:439)
    at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:924)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:781)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:220)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:43)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:545)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:43)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
    at java.lang.Thread.run(Thread.java:722)
Caused by: edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
java.lang.NullPointerException
    at edu.uci.ics.hyracks.control.cc.dataset.DatasetDirectoryService.updatedRecords(DatasetDirectoryService.java:222)
    at edu.uci.ics.hyracks.control.cc.dataset.DatasetDirectoryService.getResultPartitionLocations(DatasetDirectoryService.java:167)
    at edu.uci.ics.hyracks.control.cc.work.GetResultPartitionLocationsWork$1.run(GetResultPartitionLocationsWork.java:53)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    ... 1 more
Caused by: java.lang.NullPointerException
    at edu.uci.ics.asterix.common.context.DatasetLifecycleManager.evictCandidateDataset(DatasetLifecycleManager.java:168)
    at edu.uci.ics.asterix.common.context.DatasetLifecycleManager.open(DatasetLifecycleManager.java:144)
    at edu.uci.ics.hyracks.storage.am.common.dataflow.IndexDataflowHelper.open(IndexDataflowHelper.java:108)
    at edu.uci.ics.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.open(IndexSearchOperatorNodePushable.java:80)
    at edu.uci.ics.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.open(EmptyTupleSourceRuntimeFactory.java:50)
    at edu.uci.ics.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:104)
    at edu.uci.ics.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:81)
    at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:228)
    ... 3 more

Hi Sattam.

thanks for looking into this. We worked on this more today and we believe that 
we have identified the underlying reason for the failure and filed 
https://code.google.com/p/asterixdb/issues/detail?id=553 for this. 
However, this failure seems to be a follow-up failure and it should still be 
reproducible. To produce this failure, we used the TPC-H queries q2 and q3 as 
they are available in the regression tests. We loaded the TPC-H data at scale 
factor 1 using the attached AQL statements. The data can be generated using the 
dbgen tool from the TPC-H benchmark. I will also provide you with a location of 
the generated data files off-list.
One thing to note that this failure happened when we ran on a 3 node cluster. 
As we saw quite different behavior between running q2 on a 1 node cluster or on 
a 3 node cluster today, it might be difficult to reproduce this error on single 
machine.

Cheers,
Till

Original issue reported on code.google.com by salsuba...@gmail.com on 6 Jul 2013 at 4:29

[GoogleCodeExporter]

Attaching files

Original comment by salsuba...@gmail.com on 6 Jul 2013 at 4:31

Attachments:

• create-collections.aql
• load-data-1.aql

[GoogleCodeExporter]

Original comment by salsuba...@gmail.com on 7 Jul 2013 at 4:49

• Changed state: UnderReview

[GoogleCodeExporter]

Original comment by salsuba...@gmail.com on 8 Jul 2013 at 9:26

• Changed state: Fixed