Second and subsequent queries to Metadata.Dataset fail

[GoogleCodeExporter]

Query Q:
for $x in dataset Metadata.Dataset
return $x

What steps will reproduce the problem?
1. Start a new instance of Asterix, with clean directories etc.
2. Execute query Q and see that everything works as expected.
3. Restart the asterix instance (managix start and managix stop). No change in 
binaries or anything. Plain restart.
4. Execute query Q again to see an error

What is the expected output?
Metadata.Dataset entries as expected

What do you see instead?
Error. Stack trace from our new shiny error reporting UI :)

edu.uci.ics.asterix.metadata.MetadataException: 
edu.uci.ics.hyracks.api.exceptions.HyracksDataException: Failed to open index 
with resource ID 0 since it does not exist.
    at edu.uci.ics.asterix.metadata.MetadataNode.getFunction(MetadataNode.java:805)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
    at sun.rmi.transport.Transport$1.run(Transport.java:177)
    at sun.rmi.transport.Transport$1.run(Transport.java:174)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
    at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:273)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:251)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:160)
    at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:194)
    at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:148)
    at com.sun.proxy.$Proxy13.getFunction(Unknown Source)
    at edu.uci.ics.asterix.metadata.MetadataManager.getFunction(MetadataManager.java:528)
    at edu.uci.ics.asterix.aql.rewrites.AqlRewriter.lookupUserDefinedFunctionDecl(AqlRewriter.java:189)
    at edu.uci.ics.asterix.aql.rewrites.AqlRewriter.buildOtherUdfs(AqlRewriter.java:167)
    at edu.uci.ics.asterix.aql.rewrites.AqlRewriter.inlineDeclaredUdfs(AqlRewriter.java:142)
    at edu.uci.ics.asterix.aql.rewrites.AqlRewriter.rewrite(AqlRewriter.java:111)
    at edu.uci.ics.asterix.api.common.APIFramework.reWriteQuery(APIFramework.java:185)
    at edu.uci.ics.asterix.aql.translator.AqlTranslator.rewriteCompileQuery(AqlTranslator.java:1284)
    at edu.uci.ics.asterix.aql.translator.AqlTranslator.handleQuery(AqlTranslator.java:1383)
    at edu.uci.ics.asterix.aql.translator.AqlTranslator.compileAndExecute(AqlTranslator.java:278)
    at edu.uci.ics.asterix.api.http.servlet.APIServlet.doPost(APIServlet.java:85)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:546)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:483)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:970)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:411)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:904)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110)
    at org.eclipse.jetty.server.Server.handle(Server.java:347)
    at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:439)
    at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:924)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:781)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:220)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:43)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:545)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:43)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
    at java.lang.Thread.run(Thread.java:722)
Caused by: edu.uci.ics.hyracks.api.exceptions.HyracksDataException: Failed to 
open index with resource ID 0 since it does not exist.
    at edu.uci.ics.hyracks.storage.am.common.dataflow.IndexLifecycleManager.open(IndexLifecycleManager.java:84)
    at edu.uci.ics.asterix.metadata.MetadataNode.searchIndex(MetadataNode.java:941)
    at edu.uci.ics.asterix.metadata.MetadataNode.getFunction(MetadataNode.java:798)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
    at sun.rmi.transport.Transport$1.run(Transport.java:177)
    at sun.rmi.transport.Transport$1.run(Transport.java:174)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    ... 1 more

Please use labels and text to provide additional information.
hyracks SHA: 68bd0ca0ea8f386aa076f90549cb32e3456425f8
asterixdb SHA: fc3f504366be25f89bebba4fca73f2942dd55448

Raman, I am not sure where this is coming from. Since it involves Metadata, 
assigning it to you.

Original issue reported on code.google.com by madhusudancs on 2 Jun 2013 at 9:38

[GoogleCodeExporter]

I'll handle this one.

Original comment by salsuba...@gmail.com on 2 Jun 2013 at 11:25

[GoogleCodeExporter]

Some more information to help analyze the issue. Querying an internal dataset 
also leads to the Exception, and the issue is not restricted to querying 
metadata dataset.

Here are the steps to reproduce;

1. create asterix instance using managix.
2. run the AQL statements in the attached file. 
3. stop asterix instance using managix (the same instance that was created in 
step 1)
4. start asterix instance using managix (the instance that was stopped in step 
3)
5. From a new Web UI session, run this select query

use dataverse test;

for $l in dataset Employee 
return $l

You will see the Exception on Web UI

edu.uci.ics.asterix.metadata.MetadataException: 
edu.uci.ics.asterix.metadata.MetadataException: 
edu.uci.ics.hyracks.api.exceptions.HyracksDataException: Failed to open index 
with resource ID 0 since it does not exist.
 caused by: edu.uci.ics.asterix.aql.translator.AqlTranslator.compileAndExecute(AqlTranslator.java:294)
 caused by: edu.uci.ics.asterix.aql.translator.AqlTranslator.handleUseDataverseStatement(AqlTranslator.java:338)
 caused by: edu.uci.ics.asterix.metadata.MetadataNode.getDataverse(MetadataNode.java:614)
 caused by: edu.uci.ics.hyracks.storage.am.common.dataflow.IndexLifecycleManager.open(IndexLifecycleManager.java:84)

Original comment by khfaraaz82 on 3 Jun 2013 at 7:47

Attachments:

• create-dt-insert-dt.txt

[GoogleCodeExporter]

Original comment by salsuba...@gmail.com on 3 Jun 2013 at 5:41

• Changed state: UnderReview

[GoogleCodeExporter]

Original comment by salsuba...@gmail.com on 3 Jun 2013 at 6:47

• Changed state: Fixed