search

CarnationWang23 / hyracks

Automatically exported from code.google.com/p/hyracks
Apache License 2.0
0 stars 0 forks source link

Exception when querying large data for VXQuery. #141

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago 
What steps will reproduce the problem?
1. Run VXQuery on an 8 node cluster using +500G dataset with a large join.
2. During the probe phase of the join, the following error is thrown.
3.

What is the expected output? What do you see instead?

Query should have completed with a result instead of throwing an exception.
1 node has no error.
1 node shows the following exception
--------------------------------------------
edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
edu.uci.ics.hyracks.api.exceptions.Hyracks
DataException: java.lang.InterruptedException
    at edu.uci.ics.hyracks.control.nc.Task.pushFrames(Task.java:342)
    at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:274)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: edu.uci.ics.hyracks.api.exceptions.HyracksDataException: 
java.lang.InterruptedException
    at edu.uci.ics.hyracks.dataflow.std.collectors.NonDeterministicChannelReader.findNextSender
(NonDeterministicChannelReader.java:107)
    at edu.uci.ics.hyracks.dataflow.std.collectors.NonDeterministicFrameReader.nextFrame(NonDet
erministicFrameReader.java:38)
    at edu.uci.ics.hyracks.control.nc.Task.pushFrames(Task.java:321)
    ... 4 more
Caused by: java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:503)
    at edu.uci.ics.hyracks.dataflow.std.collectors.NonDeterministicChannelReader.findNextSender
(NonDeterministicChannelReader.java:105)
    ... 6 more
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Abstrac
tQueuedSynchronizer.java:1301)
    at java.util.concurrent.Semaphore.acquire(Semaphore.java:472)
    at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:276)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Abstrac
tQueuedSynchronizer.java:1301)
    at java.util.concurrent.Semaphore.acquire(Semaphore.java:472)
    at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:276)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Abstrac
tQueuedSynchronizer.java:1301)
    at java.util.concurrent.Semaphore.acquire(Semaphore.java:472)
    at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:276)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Abstrac
tQueuedSynchronizer.java:1301)
    at java.util.concurrent.Semaphore.acquire(Semaphore.java:472)
    at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:276)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

6 nodes show the following exception
--------------------------------------------
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Abstrac
tQueuedSynchronizer.java:1301)
    at java.util.concurrent.Semaphore.acquire(Semaphore.java:472)
    at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:276)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Abstrac
tQueuedSynchronizer.java:1301)
    at java.util.concurrent.Semaphore.acquire(Semaphore.java:472)
    at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:276)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.
java:1301)
    at java.util.concurrent.Semaphore.acquire(Semaphore.java:472)
    at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:276)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.
java:1301)
    at java.util.concurrent.Semaphore.acquire(Semaphore.java:472)
    at edu.uci.ics.hyracks.control.nc.Task.run(Task.java:276)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Dec 11, 2014 1:20:48 AM 
edu.uci.ics.hyracks.control.common.dataset.ResultStateSweeper run

Please use labels and text to provide additional information.

Original issue reported on code.google.com by ecarm...@ucr.edu on 11 Dec 2014 at 6:53

GoogleCodeExporter commented 8 years ago 
Hi Preston,

   InterruptedException usually is not the root cause.  It is because the job is terminated by a task failure event sent from a slave to the master.   Can you check the exception at the client side (where you called hyracksClient.waitForCompletion(...)) -- that usually is the root cause?      

Yingyi

Original comment by buyingyi@gmail.com on 11 Dec 2014 at 7:04