Race condition during record casting

[GoogleCodeExporter]

What steps will reproduce the problem?

Unzip attached archive into asterixdb source code root and run 
asterix-app/src/test/java/edu/uci/ics/asterix/test/runtime/RepeatedTest.java 
JUnit test.
Bug is non-deterministic, hence the test might run for a while but eventually 
will hit an ArrayIndexOutOfBoundsException:

SEVERE: Job failed on account of:
edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
edu.uci.ics.hyracks.algebricks.common.exceptions.AlgebricksException: 
edu.uci.ics.asterix.common.exceptions.AsterixException: 
java.lang.IllegalArgumentException: java.lang.ArrayIndexOutOfBoundsException: 33

edu.uci.ics.hyracks.api.exceptions.HyracksException: Job failed on account of:
edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
edu.uci.ics.hyracks.algebricks.common.exceptions.AlgebricksException: 
edu.uci.ics.asterix.common.exceptions.AsterixException: 
java.lang.IllegalArgumentException: java.lang.ArrayIndexOutOfBoundsException: 33

    at edu.uci.ics.hyracks.control.cc.job.JobRun.waitForCompletion(JobRun.java:207)
    at edu.uci.ics.hyracks.control.cc.work.WaitForJobCompletionWork$1.run(WaitForJobCompletionWork.java:44)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:722)
Caused by: edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
edu.uci.ics.hyracks.algebricks.common.exceptions.AlgebricksException: 
edu.uci.ics.asterix.common.exceptions.AsterixException: 
java.lang.IllegalArgumentException: java.lang.ArrayIndexOutOfBoundsException: 33
    at edu.uci.ics.hyracks.control.common.utils.ExceptionUtils.setNodeIds(ExceptionUtils.java:41)
    at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:291)
    ... 3 more
Caused by: edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
edu.uci.ics.hyracks.algebricks.common.exceptions.AlgebricksException: 
edu.uci.ics.asterix.common.exceptions.AsterixException: 
java.lang.IllegalArgumentException: java.lang.ArrayIndexOutOfBoundsException: 33
    at edu.uci.ics.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.produceTuple(AssignRuntimeFactory.java:146)
    at edu.uci.ics.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.nextFrame(AssignRuntimeFactory.java:121)
    at edu.uci.ics.hyracks.dataflow.common.comm.util.FrameUtils.flushFrame(FrameUtils.java:38)
    at edu.uci.ics.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.appendProjectionToFrame(AbstractOneInputOneOutputOneFramePushRuntime.java:80)
    at edu.uci.ics.hyracks.algebricks.runtime.operators.std.StreamProjectRuntimeFactory$1.nextFrame(StreamProjectRuntimeFactory.java:76)
    at edu.uci.ics.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$2.nextFrame(AlgebricksMetaOperatorDescriptor.java:136)
    at edu.uci.ics.hyracks.dataflow.common.comm.util.FrameUtils.flushFrame(FrameUtils.java:38)
    at edu.uci.ics.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:149)
    at edu.uci.ics.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.open(EmptyTupleSourceRuntimeFactory.java:56)
    at edu.uci.ics.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:104)
    at edu.uci.ics.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:81)
    at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:234)
    ... 3 more
Caused by: 
edu.uci.ics.hyracks.algebricks.common.exceptions.AlgebricksException: 
edu.uci.ics.asterix.common.exceptions.AsterixException: 
java.lang.IllegalArgumentException: java.lang.ArrayIndexOutOfBoundsException: 33
    at edu.uci.ics.asterix.runtime.evaluators.functions.CastRecordDescriptor$2$1.evaluate(CastRecordDescriptor.java:91)
    at edu.uci.ics.hyracks.algebricks.core.algebra.expressions.LogicalExpressionJobGenToExpressionRuntimeProviderAdapter$ScalarEvaluatorFactoryAdapter$1.evaluate(LogicalExpressionJobGenToExpressionRuntimeProviderAdapter.java:106)
    at edu.uci.ics.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.produceTuple(AssignRuntimeFactory.java:144)
    ... 14 more
Caused by: edu.uci.ics.asterix.common.exceptions.AsterixException: 
java.lang.IllegalArgumentException: java.lang.ArrayIndexOutOfBoundsException: 33
    at edu.uci.ics.asterix.om.pointables.cast.ACastVisitor.visit(ACastVisitor.java:88)
    at edu.uci.ics.asterix.om.pointables.cast.ACastVisitor.visit(ACastVisitor.java:1)
    at edu.uci.ics.asterix.om.pointables.ARecordPointable.accept(ARecordPointable.java:281)
    at edu.uci.ics.asterix.runtime.evaluators.functions.CastRecordDescriptor$2$1.evaluate(CastRecordDescriptor.java:87)
    ... 16 more
Caused by: java.lang.IllegalArgumentException: 
java.lang.ArrayIndexOutOfBoundsException: 33
    at edu.uci.ics.hyracks.data.std.primitive.UTF8StringPointable.charSize(UTF8StringPointable.java:120)
    at edu.uci.ics.hyracks.data.std.primitive.UTF8StringPointable.hash(UTF8StringPointable.java:207)
    at edu.uci.ics.hyracks.data.std.accessors.PointableBinaryHashFunctionFactory$1.hash(PointableBinaryHashFunctionFactory.java:43)
    at edu.uci.ics.asterix.om.types.ARecordType.findFieldPosition(ARecordType.java:144)
    at edu.uci.ics.asterix.builders.RecordBuilder.addField(RecordBuilder.java:167)
    at edu.uci.ics.asterix.om.pointables.cast.ARecordCaster.writeOutput(ARecordCaster.java:316)
    at edu.uci.ics.asterix.om.pointables.cast.ARecordCaster.castRecord(ARecordCaster.java:127)
    at edu.uci.ics.asterix.om.pointables.cast.ACastVisitor.visit(ACastVisitor.java:86)
    ... 19 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 33
    at edu.uci.ics.hyracks.data.std.primitive.UTF8StringPointable.charSize(UTF8StringPointable.java:100)
    ... 26 more

The exception happens while job tries to write the casted (open) record to the 
output buffer. To perform this write field name as well as it's value must be 
written. When the record caster tries to read field name from uncasted(closed) 
record it fails due to the fact that buffer, containing field name somehow 
modified and offset in the buffer is incorrect.

Any help will be appreciated, since at this point I am running out of ideas 
what is the cause of the problem 

Original issue reported on code.google.com by ildar.absalyamov on 7 May 2014 at 7:13

Attachments:

• asterix-bug.tar.gz

[GoogleCodeExporter]

Ildar. which branch is this code in? I want to take a look at it.

Original comment by bamou...@gmail.com on 8 May 2014 at 2:32

[GoogleCodeExporter]

It's in master

Original comment by ildar.absalyamov on 8 May 2014 at 4:59

[GoogleCodeExporter]

Original comment by ildar.absalyamov on 8 May 2014 at 10:55

[GoogleCodeExporter]

Original comment by ildar.absalyamov on 22 May 2014 at 1:06

• Changed state: FixedInBranch

[GoogleCodeExporter]

Original comment by ildar.absalyamov on 23 May 2014 at 5:56

• Changed state: Fixed