search

lwhay / asterixdb

Automatically exported from code.google.com/p/asterixdb
0 stars 0 forks source link

Index NLJ breaks as it tries to use secondary index #701

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
-) Overview
Running an Index-Join query on our synthetic social data, query breaks by NPE 
(check the stack trace below).
(for more info on our social data schema, check 
http://asterixdb.ics.uci.edu/documentation/aql/primer.html )

-) Setting. 
Running asterix on cluster with 5 NCs, and 15 partitions (IO-devices, 3 per 
node). Scale factor of generated data is 420GB.

-) Observation 1. 
This issue occurs sporadically. Trying to reproduce it, it does not happen all 
the time.

-) Observation 2.
The history of the queries that had been run before this query matters. If 
secondary index is already open (exploited by some earlier query in healthy 
way), the issue seems not to occur.

-) Query
//A secondary index is already created on FacebookMessages.author-id
//id is the primary key for FacebookUsers

for $user in dataset('FacebookUsers')
for $message in dataset('FacebookMessages')
where $message.author-id /*+ indexnl */  = $user.id and
$user.user-since >=
datetime('2010-07-12T09:23:05')
and
$user.user-since <
datetime('2010-07-12T09:25:05')
return {
"uname": $user.name,
"message": $message.message
}

-) NC log sample:

java.lang.IllegalArgumentException: Negative position
    at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:662)
    at edu.uci.ics.hyracks.control.nc.io.IOManager.syncRead(IOManager.java:112)
    at edu.uci.ics.hyracks.storage.common.buffercache.BufferCache.read(BufferCache.java:373)
    at edu.uci.ics.hyracks.storage.common.buffercache.BufferCache.pin(BufferCache.java:154)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:600)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:631)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree.search(BTree.java:197)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree.access$600(BTree.java:69)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree$BTreeAccessor.search(BTree.java:909)
    at edu.uci.ics.hyracks.storage.am.lsm.btree.impls.LSMBTreeRangeSearchCursor.open(LSMBTreeRangeSearchCursor.java:206)
    at edu.uci.ics.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.open(LSMBTreeSearchCursor.java:77)
    at edu.uci.ics.hyracks.storage.am.lsm.btree.impls.LSMBTree.search(LSMBTree.java:357)
    at edu.uci.ics.hyracks.storage.am.lsm.common.impls.LSMHarness.search(LSMHarness.java:242)
    at edu.uci.ics.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.search(LSMTreeIndexAccessor.java:95)
    at edu.uci.ics.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:137)
    at edu.uci.ics.hyracks.control.nc.Task.pushFrames(Task.java:316)
    at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:269)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
java.lang.NullPointerException
    at edu.uci.ics.hyracks.control.nc.Task.pushFrames(Task.java:332)
    at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:269)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
java.lang.NullPointerException
    at edu.uci.ics.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:155)
    at edu.uci.ics.hyracks.control.nc.Task.pushFrames(Task.java:323)
    ... 4 more
Caused by: java.lang.NullPointerException
    at edu.uci.ics.hyracks.storage.am.lsm.common.impls.LSMIndexSearchCursor.close(LSMIndexSearchCursor.java:116)
    at edu.uci.ics.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.close(LSMBTreeSearchCursor.java:93)
    at edu.uci.ics.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:153)
    ... 5 more
java.lang.IllegalArgumentException: Negative position
    at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:662)
    at edu.uci.ics.hyracks.control.nc.io.IOManager.syncRead(IOManager.java:112)
    at edu.uci.ics.hyracks.storage.common.buffercache.BufferCache.read(BufferCache.java:373)
    at edu.uci.ics.hyracks.storage.common.buffercache.BufferCache.pin(BufferCache.java:154)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:600)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:631)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree.search(BTree.java:197)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree.access$600(BTree.java:69)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree$BTreeAccessor.search(BTree.java:909)
    at edu.uci.ics.hyracks.storage.am.lsm.btree.impls.LSMBTreeRangeSearchCursor.open(LSMBTreeRangeSearchCursor.java:206)
    at edu.uci.ics.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.open(LSMBTreeSearchCursor.java:77)
    at edu.uci.ics.hyracks.storage.am.lsm.btree.impls.LSMBTree.search(LSMBTree.java:357)
    at edu.uci.ics.hyracks.storage.am.lsm.common.impls.LSMHarness.search(LSMHarness.java:242)
    at edu.uci.ics.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.search(LSMTreeIndexAccessor.java:95)
    at edu.uci.ics.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:137)
    at edu.uci.ics.hyracks.control.nc.Task.pushFrames(Task.java:316)
    at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:269)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
java.lang.NullPointerException
    at edu.uci.ics.hyracks.control.nc.Task.pushFrames(Task.java:332)
    at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:269)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
java.lang.NullPointerException
    at edu.uci.ics.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:155)
    at edu.uci.ics.hyracks.control.nc.Task.pushFrames(Task.java:323)
    ... 4 more
Caused by: java.lang.NullPointerException
    at edu.uci.ics.hyracks.storage.am.lsm.common.impls.LSMIndexSearchCursor.close(LSMIndexSearchCursor.java:116)
    at edu.uci.ics.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.close(LSMBTreeSearchCursor.java:93)
    at edu.uci.ics.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:153)
    ... 5 more
java.lang.IllegalArgumentException: Negative position
    at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:662)
    at edu.uci.ics.hyracks.control.nc.io.IOManager.syncRead(IOManager.java:112)
    at edu.uci.ics.hyracks.storage.common.buffercache.BufferCache.read(BufferCache.java:373)
    at edu.uci.ics.hyracks.storage.common.buffercache.BufferCache.pin(BufferCache.java:154)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:600)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:631)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree.search(BTree.java:197)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree.access$600(BTree.java:69)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree$BTreeAccessor.search(BTree.java:909)
    at edu.uci.ics.hyracks.storage.am.lsm.btree.impls.LSMBTreeRangeSearchCursor.open(LSMBTreeRangeSearchCursor.java:206)
    at edu.uci.ics.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.open(LSMBTreeSearchCursor.java:77)
    at edu.uci.ics.hyracks.storage.am.lsm.btree.impls.LSMBTree.search(LSMBTree.java:357)
    at edu.uci.ics.hyracks.storage.am.lsm.common.impls.LSMHarness.search(LSMHarness.java:242)
    at edu.uci.ics.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.search(LSMTreeIndexAccessor.java:95)
    at edu.uci.ics.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:137)
    at edu.uci.ics.hyracks.control.nc.Task.pushFrames(Task.java:316)
    at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:269)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
edu.uci.ics.hyracks.storage.am.btree.exceptions.BTreeException: 
java.lang.IllegalArgumentException: Negative position
    at edu.uci.ics.hyracks.control.nc.Task.pushFrames(Task.java:332)
    at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:269)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
edu.uci.ics.hyracks.storage.am.btree.exceptions.BTreeException: 
java.lang.IllegalArgumentException: Negative position
    at edu.uci.ics.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:141)
    at edu.uci.ics.hyracks.control.nc.Task.pushFrames(Task.java:316)
    ... 4 more
Caused by: edu.uci.ics.hyracks.storage.am.btree.exceptions.BTreeException: 
java.lang.IllegalArgumentException: Negative position
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:784)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree.search(BTree.java:197)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree.access$600(BTree.java:69)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree$BTreeAccessor.search(BTree.java:909)
    at edu.uci.ics.hyracks.storage.am.lsm.btree.impls.LSMBTreeRangeSearchCursor.open(LSMBTreeRangeSearchCursor.java:206)
    at edu.uci.ics.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.open(LSMBTreeSearchCursor.java:77)
    at edu.uci.ics.hyracks.storage.am.lsm.btree.impls.LSMBTree.search(LSMBTree.java:357)
    at edu.uci.ics.hyracks.storage.am.lsm.common.impls.LSMHarness.search(LSMHarness.java:242)
    at edu.uci.ics.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.search(LSMTreeIndexAccessor.java:95)
    at edu.uci.ics.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:137)
    ... 5 more
Caused by: java.lang.IllegalArgumentException: Negative position
    at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:662)
    at edu.uci.ics.hyracks.control.nc.io.IOManager.syncRead(IOManager.java:112)
    at edu.uci.ics.hyracks.storage.common.buffercache.BufferCache.read(BufferCache.java:373)
    at edu.uci.ics.hyracks.storage.common.buffercache.BufferCache.pin(BufferCache.java:154)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:600)
    at edu.uci.ics.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:631)
    ... 14 more

Original issue reported on code.google.com by pouria.p...@gmail.com on 24 Jan 2014 at 5:24

GoogleCodeExporter commented 9 years ago 
Zach - maybe you can own this one?  (Seems like LSM-as-secondary index, and 
index lifecycle, know-how will be crucial.)  Thx!

Original comment by dtab...@gmail.com on 27 Jan 2014 at 8:47

GoogleCodeExporter commented 9 years ago 
Is this issue blocking you Pouria?

Also, would it be possible to get a more minimal reproducer? Mike's thoughts 
(which I agree with) are that this is probably not related to parallelism so a 
single node reproducer might be possible--including using a much smaller data 
volume.

If it is not urgent then I'd prefer to wait until we complete the LSM 
experiments before looking at this issue since the deadline is approaching fast.

Original comment by zheilb...@gmail.com on 29 Jan 2014 at 6:46

GoogleCodeExporter commented 9 years ago 
Not a blocking issue at this point. As I mentioned in "Observation 2" in 
issue's description, It is possible to execute the query successfully.

I will try to reproduce it on a smaller setup ...

Original comment by pouria.p...@gmail.com on 29 Jan 2014 at 6:59

GoogleCodeExporter commented 9 years ago 
To be clear on my thought: You can only get us to do an index NL join if you 
"hint" that we should.  My guess is that we have VERY few test cases, so what's 
happening could be something very, very basic that we simply have not tested 
before.  (This was fairly untrodden code until recently.)

Original comment by dtab...@gmail.com on 29 Jan 2014 at 10:28