search

bulldog2011 / bigqueue

A big, fast and persistent queue based on memory mapped file.
http://bulldog2011.github.com/bigqueue
Apache License 2.0
556 stars 219 forks source link

Getting an IndexOutOfBoundException when IBigQueue.peek #24

Open gimantha opened 8 years ago

gimantha commented 8 years ago

Hi @bulldog2011, @TobiasMende , @wjw465150 , @illya13

We use bigqueue implementation to buffer incoming data in our code base. We use a background thread to consume the queue (every 1.5 seconds). Recently we did some load test and published some data. We could observe the following exception occurred when we restart our server.

[2016-08-02 10:37:28,676] ERROR {org.wso2.carbon.analytics.dataservice.core.indexing.AnalyticsDataIndexer} - Error in processing index batch operations: null java.lang.IndexOutOfBoundsException at com.leansoft.bigqueue.BigArrayImpl.validateIndex(BigArrayImpl.java:458) at com.leansoft.bigqueue.BigArrayImpl.get(BigArrayImpl.java:399) at com.leansoft.bigqueue.BigQueueImpl.peek(BigQueueImpl.java:144) at org.wso2.carbon.analytics.dataservice.core.indexing.LocalIndexDataStore$LocalIndexDataQueue.peekNext(LocalIndexDataStore.java:269) I would like to know, under what conditions, the above exception can be thrown? We use the bigqueue implementation here[1]

Appreciate your help!

[1] https://github.com/wso2/carbon-analytics/blob/master/components/analytics-core/org.wso2.carbon.analytics.dataservice.core/src/main/java/org/wso2/carbon/analytics/dataservice/core/indexing/LocalIndexDataStore.java

Thanks,

jsabin commented 6 years ago

Seeing the same problem. Anyone know the cause or how to prevent this? Seen this on version 0.7.0.

java.lang.IndexOutOfBoundsException         at com.leansoft.bigqueue.BigArrayImpl.validateIndex(BigArrayImpl.java:458)         at com.leansoft.bigqueue.BigArrayImpl.get(BigArrayImpl.java:399)         at com.leansoft.bigqueue.BigQueueImpl.dequeue(BigQueueImpl.java:104)         at com.proofpoint.queue.FileBackedQueue.dequeue(FileBackedQueue.java:183)

MrWickedG commented 5 years ago

Does anybody have any solution for this issue? Please let me know @jsabin @gimantha

gimantha commented 5 years ago

How big is the content that you are storing in the queue?

On Mon, Nov 5, 2018 at 3:26 PM MrWickedG notifications@github.com wrote:

Does anybody have any solution for this issue? Please let me know

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/boboweike/bigqueue/issues/24#issuecomment-435817055, or mute the thread https://github.com/notifications/unsubscribe-auth/AI7bX7Fz4muhjcUPDGB1BEkC2SYUm1ewks5usAtDgaJpZM4JaWEU .

-- Gimantha Bandara | Associate Technical Lead | WSO2 Inc. (m) +94714961919 | (w) +94117435800 | (e) gimantha@wso2.com GET INTEGRATION AGILE Integration Agility for Digitally Driven Business

MrWickedG commented 5 years ago

@gimantha We create/delete 32 mb page files every several minutes, so i'd say its quite big while maintaining close to a hundred of queues. This error is quite occasional but we did not find a solution for it yet.

Did you solve it somehow?

gimantha commented 5 years ago

Unfortunately no. I was not able to reproduce the issue now.. :(

On Monday, November 5, 2018, MrWickedG notifications@github.com wrote:

@gimantha https://github.com/gimantha We create/delete 32 mb page files every several minutes, so i'd say its quite big while maintaining close to a hundred of queues. This error is quite occasional but we did not find a solution for it yet.

Did you solve it somehow?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/boboweike/bigqueue/issues/24#issuecomment-435838875, or mute the thread https://github.com/notifications/unsubscribe-auth/AI7bX5rBSLN_goor23uf4GHI0-9Cgu9hks5usB2EgaJpZM4JaWEU .

-- Gimantha Bandara | Associate Technical Lead | WSO2 Inc. (m) +94714961919 | (w) +94117435800 | (e) gimantha@wso2.com GET INTEGRATION AGILE Integration Agility for Digitally Driven Business

yoonforh commented 4 years ago

i found the problem in BigQueueImpl.java.

in removeAll method of the BigQueueImpl.java current code is 

            queueFrontWriteLock.lock();
            this.innerArray.removeAll();
            this.queueFrontIndex.set(0L);

i think the code should be rearranged like this

            queueFrontWriteLock.lock();
            // if some thread access queueFrontIndex.get() without queueFrontWriteLock, 
            // then this could be a problem.
            // you should clear queueFrontIndex first (peek or dequeue)
            // before remove all the innerArray
            this.queueFrontIndex.set(0L);
            this.innerArray.removeAll();

this code can cause "java.lang.IndexOutOfBoundsException: null" in peek() method and in dequeue() method. i found the dequeue() method scenario can happen when the gc() method is invoked at the time.

if some thread access queueFrontIndex.get() without queueFrontWriteLock, then this could be a problem. if old version of removeAll() method is called and gc() is called after innerArray.removeAll() and before queueFrontIndex.set(0), then queueFrontIndex will return some bigger value and removeBeforeIndex will be called on that index. and later, dequeue will be called and queueFrontIndex will be zero but the innerArray.arrayTailIndex will be set to the some positive value. yes, innerArray.validateIndex(0) will cause a problem!