Closed GoogleCodeExporter closed 9 years ago
Is it metadata only?
Original comment by buyingyi@gmail.com
on 6 Jun 2013 at 3:23
For now, I tried this concurrency test to scan Metadata Datasets only. I have
not tried this concurrency test program on internal datasets. It can be used to
run concurrent scans on internal datasets too, those that are created by users..
PS: Thanks to Madhusudhan for having reviewed and helped me in refining my
concurrency test program.
Original comment by khfaraaz82
on 6 Jun 2013 at 3:39
Can you add the test as part of the build tests in your private branch?
Original comment by buyingyi@gmail.com
on 6 Jun 2013 at 3:42
Yes, I can do that. Addressing review comments for my thesis now.
I will definitely add this test to my branch, once those comments are addressed.
Original comment by khfaraaz82
on 6 Jun 2013 at 3:45
Similar NPEs are also seen when scanning Index and Dataverse metadata datasets.
Jun 05, 2013 8:03:05 PM
edu.uci.ics.hyracks.algebricks.core.rewriter.base.HeuristicOptimizer optimize
INFO: Optimized Plan:
distribute result [%0->$$0]
-- DISTRIBUTE_RESULT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$0])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
data-scan []<-[$$2, $$0] <- Metadata:Dataverse
-- DATASOURCE_SCAN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
Jun 05, 2013 8:03:05 PM edu.uci.ics.asterix.api.http.servlet.RESTAPIServlet
doGet
SEVERE: null
java.lang.NullPointerException
Jun 05, 2013 8:03:05 PM edu.uci.ics.asterix.api.http.servlet.RESTAPIServlet
doGet
SEVERE: null
java.lang.NullPointerException
at edu.uci.ics.asterix.formats.nontagged.AqlSerializerDeserializerProvider.getSerializerDeserializer(AqlSerializerDeserializerProvider.java:72)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.JobGenHelper.mkRecordDescriptor(JobGenHelper.java:58)
at edu.uci.ics.asterix.metadata.declared.AqlMetadataProvider.buildBtreeRuntime(AqlMetadataProvider.java:515)
at edu.uci.ics.asterix.metadata.declared.AqlMetadataProvider.buildInternalDatasetScan(AqlMetadataProvider.java:326)
at edu.uci.ics.asterix.metadata.declared.AqlMetadataProvider.getScannerRuntime(AqlMetadataProvider.java:303)
at edu.uci.ics.hyracks.algebricks.core.algebra.operators.physical.DataSourceScanPOperator.contributeRuntimeOperator(DataSourceScanPOperator.java:82)
at edu.uci.ics.hyracks.algebricks.core.algebra.operators.logical.AbstractLogicalOperator.contributeRuntimeOperator(AbstractLogicalOperator.java:157)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.PlanCompiler.compileOpRef(PlanCompiler.java:91)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.PlanCompiler.compileOpRef(PlanCompiler.java:78)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.PlanCompiler.compileOpRef(PlanCompiler.java:78)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.PlanCompiler.compileOpRef(PlanCompiler.java:78)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.PlanCompiler.compileOpRef(PlanCompiler.java:78)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.PlanCompiler.compilePlan(PlanCompiler.java:54)
at edu.uci.ics.hyracks.algebricks.compiler.api.HeuristicCompilerFactoryBuilder$1$1.createJob(HeuristicCompilerFactoryBuilder.java:98)
at edu.uci.ics.asterix.api.common.APIFramework.compileQuery(APIFramework.java:351)
at edu.uci.ics.asterix.aql.translator.AqlTranslator.rewriteCompileQuery(AqlTranslator.java:1293)
at edu.uci.ics.asterix.aql.translator.AqlTranslator.handleQuery(AqlTranslator.java:1387)
at edu.uci.ics.asterix.aql.translator.AqlTranslator.compileAndExecute(AqlTranslator.java:279)
at edu.uci.ics.asterix.api.http.servlet.RESTAPIServlet.doGet(RESTAPIServlet.java:98)
at edu.uci.ics.asterix.api.http.servlet.QueryAPIServlet.doGet(QueryAPIServlet.java:25)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:546)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:483)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:970)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:411)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:904)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110)
at org.eclipse.jetty.server.Server.handle(Server.java:347)
at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:439)
at org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:907)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:562)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:43)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:545)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:43)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
at java.lang.Thread.run(Thread.java:722)
....
Jun 05, 2013 8:03:06 PM
edu.uci.ics.hyracks.algebricks.core.jobgen.impl.JobGenHelper mkRecordDescriptor
WARNING: No type for variable $$2
Jun 05, 2013 8:03:06 PM
edu.uci.ics.hyracks.algebricks.core.rewriter.base.HeuristicOptimizer optimize
INFO: Optimized Plan:
distribute result [%0->$$0]
-- DISTRIBUTE_RESULT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
project ([$$0])
-- STREAM_PROJECT |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
data-scan []<-[$$2, $$3, $$4, $$0] <- Metadata:Index
-- DATASOURCE_SCAN |PARTITIONED|
exchange
-- ONE_TO_ONE_EXCHANGE |PARTITIONED|
empty-tuple-source
-- EMPTY_TUPLE_SOURCE |PARTITIONED|
java.lang.NullPointerException
java.lang.NullPointerException
at edu.uci.ics.asterix.formats.nontagged.AqlSerializerDeserializerProvider.getSerializerDeserializer(AqlSerializerDeserializerProvider.java:72)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.JobGenHelper.mkRecordDescriptor(JobGenHelper.java:58)
at edu.uci.ics.asterix.metadata.declared.AqlMetadataProvider.buildBtreeRuntime(AqlMetadataProvider.java:515)
at edu.uci.ics.asterix.metadata.declared.AqlMetadataProvider.buildInternalDatasetScan(AqlMetadataProvider.java:326)
at edu.uci.ics.asterix.metadata.declared.AqlMetadataProvider.getScannerRuntime(AqlMetadataProvider.java:303)
at edu.uci.ics.hyracks.algebricks.core.algebra.operators.physical.DataSourceScanPOperator.contributeRuntimeOperator(DataSourceScanPOperator.java:82)
at edu.uci.ics.hyracks.algebricks.core.algebra.operators.logical.AbstractLogicalOperator.contributeRuntimeOperator(AbstractLogicalOperator.java:157)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.PlanCompiler.compileOpRef(PlanCompiler.java:91)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.PlanCompiler.compileOpRef(PlanCompiler.java:78)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.PlanCompiler.compileOpRef(PlanCompiler.java:78)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.PlanCompiler.compileOpRef(PlanCompiler.java:78)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.PlanCompiler.compileOpRef(PlanCompiler.java:78)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.PlanCompiler.compilePlan(PlanCompiler.java:54)
at edu.uci.ics.hyracks.algebricks.compiler.api.HeuristicCompilerFactoryBuilder$1$1.createJob(HeuristicCompilerFactoryBuilder.java:98)
at edu.uci.ics.asterix.api.common.APIFramework.compileQuery(APIFramework.java:351)
at edu.uci.ics.asterix.aql.translator.AqlTranslator.rewriteCompileQuery(AqlTranslator.java:1293)
at edu.uci.ics.asterix.aql.translator.AqlTranslator.handleQuery(AqlTranslator.java:1387)
at edu.uci.ics.asterix.aql.translator.AqlTranslator.compileAndExecute(AqlTranslator.java:279)
at edu.uci.ics.asterix.api.http.servlet.RESTAPIServlet.doGet(RESTAPIServlet.java:98)
at edu.uci.ics.asterix.api.http.servlet.QueryAPIServlet.doGet(QueryAPIServlet.java:25)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:546)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:483)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:970)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:411)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:904)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110)
at org.eclipse.jetty.server.Server.handle(Server.java:347)
at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:439)
at org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:907)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:562)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:43)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:545)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:43)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
at java.lang.Thread.run(Thread.java:722)
Original comment by khfaraaz82
on 6 Jun 2013 at 3:46
Concurrent scans of internal datasets also leads to NPE as reported above. The
stack trace for the NPE in the cc log is exactly similar to the one above. To
reproduce, please run these AQL statements from Web UI and then run the
WisconsinScan.java program from Eclipse, and you will see the NPE in cc.log
under managix directory.
The data is generated as per Wisconsin benchmark and the queries used to run
concurrently are also from that benchmark.
drop dataverse test if exists;
create dataverse test;
use dataverse test;
create type Schema as closed {
unique1: int32,
unique2: int32,
two: int32,
four: int32,
ten: int32,
twenty: int32,
onePercent: int32,
tenPercent: int32,
twentyPercent: int32,
fiftyPercent: int32,
unique3: int32,
evenOnePercent: int32,
oddOnePercent: int32,
stringu1: string,
stringu2: string,
string4: string
}
create dataset tenktup1(Schema) primary key unique2 ;
load dataset tenktup1 using localfs
(("path"="localhost:///home/kfmohamm/data-gen/one-mil-tup.adm"),("format"="delim
ited-text"),("delimiter"="|")) pre-sorted;
Here is the stack trace from cc.log
Jun 07, 2013 5:47:59 PM
edu.uci.ics.hyracks.algebricks.core.jobgen.impl.JobGenHelper mkRecordDescriptor
WARNING: No type for variable $$2
java.lang.NullPointerException
at edu.uci.ics.asterix.formats.nontagged.AqlSerializerDeserializerProvider.getSerializerDeserializer(AqlSerializerDeserializerProvider.java:72)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.JobGenHelper.mkRecordDescriptor(JobGenHelper.java:58)
at edu.uci.ics.asterix.metadata.declared.AqlMetadataProvider.buildBtreeRuntime(AqlMetadataProvider.java:515)
at edu.uci.ics.asterix.metadata.declared.AqlMetadataProvider.buildInternalDatasetScan(AqlMetadataProvider.java:326)
at edu.uci.ics.asterix.metadata.declared.AqlMetadataProvider.getScannerRuntime(AqlMetadataProvider.java:303)
at edu.uci.ics.hyracks.algebricks.core.algebra.operators.physical.DataSourceScanPOperator.contributeRuntimeOperator(DataSourceScanPOperator.java:82)
at edu.uci.ics.hyracks.algebricks.core.algebra.operators.logical.AbstractLogicalOperator.contributeRuntimeOperator(AbstractLogicalOperator.java:157)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.PlanCompiler.compileOpRef(PlanCompiler.java:91)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.PlanCompiler.compileOpRef(PlanCompiler.java:78)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.PlanCompiler.compileOpRef(PlanCompiler.java:78)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.PlanCompiler.compileOpRef(PlanCompiler.java:78)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.PlanCompiler.compileOpRef(PlanCompiler.java:78)
at edu.uci.ics.hyracks.algebricks.core.jobgen.impl.PlanCompiler.compilePlan(PlanCompiler.java:54)
at edu.uci.ics.hyracks.algebricks.compiler.api.HeuristicCompilerFactoryBuilder$1$1.createJob(HeuristicCompilerFactoryBuilder.java:98)
at edu.uci.ics.asterix.api.common.APIFramework.compileQuery(APIFramework.java:351)
at edu.uci.ics.asterix.aql.translator.AqlTranslator.rewriteCompileQuery(AqlTranslator.java:1293)
at edu.uci.ics.asterix.aql.translator.AqlTranslator.handleQuery(AqlTranslator.java:1387)
at edu.uci.ics.asterix.aql.translator.AqlTranslator.compileAndExecute(AqlTranslator.java:279)
at edu.uci.ics.asterix.api.http.servlet.RESTAPIServlet.doGet(RESTAPIServlet.java:98)
at edu.uci.ics.asterix.api.http.servlet.QueryAPIServlet.doGet(QueryAPIServlet.java:25)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:546)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:483)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:970)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:411)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:904)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110)
at org.eclipse.jetty.server.Server.handle(Server.java:347)
at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:439)
at org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:907)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:562)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:43)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:545)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:43)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
at java.lang.Thread.run(Thread.java:722)
Original comment by khfaraaz82
on 8 Jun 2013 at 1:19
Attachments:
Original comment by khfaraaz82
on 10 Jun 2013 at 6:35
Is this issue being investigated ?
Original comment by khfaraaz82
on 18 Jun 2013 at 9:59
I am yet to find the exact cause
Original comment by ram...@uci.edu
on 18 Jun 2013 at 10:09
Yingyi - this says "concurrent" - should you take this one?
Original comment by dtab...@gmail.com
on 27 Jun 2013 at 4:48
With the new transactions system in place now. Can someone please run this test
to verify. This was a concurrency test that was giving NPE's.
Thanks.
Original comment by khfaraaz82
on 6 Nov 2013 at 4:39
I have found the source of the issue.
The source is as follows.
During concurrent query compilations, a datatype object cached in
MetadataManager is accessed by those compilers. The datatype object has an
ARecordType object in it and the ARecordType object has UTF8StringPointable
object in it. The problem occurs when the UTF8StringPointable.compare() is
called since the UTF8StringPointable is not stateless, so UTF8StringPointable
object in the ARecordType object (which is in the datatype object) shows the
problem when it is called concurrently.
One possible solution can be creating a new comparator for UTF8Sting which is
STATELESS.
Or, simply we can copy the datatype object in the metdataManager's cache when
the datatype object is returned to the caller.
Please your opinion or suggestions.
Young-Seok
Original comment by kiss...@gmail.com
on 20 Nov 2013 at 9:02
Original comment by kiss...@gmail.com
on 21 Nov 2013 at 10:47
Original issue reported on code.google.com by
khfaraaz82
on 6 Jun 2013 at 3:20Attachments: