EsRejectedExecutionException[rejected execution (queue capacity 50)

[rotastrain]

I started indexing after which it starts indexing does a few records and then stops. Logs show this.

MongoDBRiverBulkProcessor] Bulk processor failed. failure in bulk execution: [1]: index [events_index], type [event], id [5310c7cd78656f3509a60200], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@14287bb0]] [2]: index [events_index], type [event], id [5310c7cd78656f3509a70200], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication

I think the queue capacity is getting overloaded. Is this an issue with the river? Any solution?

tried using ES 1.0.0 and 1.0.1 .same issue.

[rotastrain]

i increased thread pool : bulk: size : 60 queue : 100 and it stopped coming. Is there any thing else i should change

[richardwilly98]

@rotastrain Which version of the river are you running? Can you please provide ES log file? There is an additional check in the latest release 2.0.0 to make sure at least 10% bulk queue is available.

[dblado]

I just attempted to upgrade to ES 1.0.1 and river 2.0.0 to fix 228 and I'm seeing this error too.

[2014-03-09 14:15:36,681][INFO ][org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor] Waiting for bulk queue to empty... [2014-03-09 14:15:36,716][INFO ][cluster.metadata ] [Penance] [vendop_v1] update_mapping vendor [2014-03-09 14:15:36,779][ERROR][org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor] Bulk processor failed. failure in bulk execution: [3]: index [vendop_v1], type [vendor], id [5226e0e1c64c8c021817ef66], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@88d013d]] [4]: index [vendop_v1], type [vendor], id [5302d4e2c61e360991327e9b], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@88d013d]] [6]: index [vendop_v1], type [vendor], id [5303aa6ac61e360991327e9e], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@88d013d]] [2014-03-09 14:15:36,779][DEBUG][river.mongodb.util ] setRiverStatus called with vendor_v1 - IMPORT_FAILED [2014-03-09 14:15:36,792][ERROR][org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor] Bulk processor failed. failure in bulk execution: [2]: index [vendop_v1], type [vendor], id [5226e1fcc64c8c021056df8e], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@576a7290]] [5]: index [vendop_v1], type [vendor], id [5226e226c64c8c0210573d9b], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@576a7290]] [6]: index [vendop_v1], type [vendor], id [5226de4bc64c8c019877da36], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@576a7290]] [9]: index [vendop_v1], type [vendor], id [53051fd0c61e360992e4af5a], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@576a7290]] [2014-03-09 14:15:36,793][DEBUG][river.mongodb.util ] setRiverStatus called with vendor_v1 - IMPORT_FAILED [2014-03-09 14:15:36,797][ERROR][org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor] Bulk processor failed. failure in bulk execution: [2]: index [vendop_v1], type [review], id [52ae6293c61e367c25730b08], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@23bff716]] [8]: index [vendop_v1], type [review], id [52ae65f8c61e367c2430b094], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@23bff716]] [9]: index [vendop_v1], type [review], id [52ae66afc61e367c25730b0d], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@23bff716]] [15]: index [vendop_v1], type [review], id [52ae6b97c61e367c2430b097], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@23bff716]] [2014-03-09 14:15:36,798][DEBUG][river.mongodb.util ] setRiverStatus called with review_v1 - IMPORT_FAILED [2014-03-09 14:15:36,805][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Closing river vendor_v1 Exception in thread "elasticsearch[Penance][mongodb_river_indexer][T#1]" org.elasticsearch.ElasticsearchIllegalStateException: bulk process already closed at org.elasticsearch.action.bulk.BulkProcessor.executeIfNeeded(BulkProcessor.java:255) at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:240) at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:234) at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:230) at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:216) at org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor.addBulkRequest(MongoDBRiverBulkProcessor.java:184) at org.elasticsearch.river.mongodb.Indexer.updateBulkRequest(Indexer.java:246) at org.elasticsearch.river.mongodb.Indexer.processBlockingQueue(Indexer.java:218) at org.elasticsearch.river.mongodb.Indexer.run(Indexer.java:73) at java.lang.Thread.run(Thread.java:724) [2014-03-09 14:15:36,807][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Status thread interrupted java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at org.elasticsearch.river.mongodb.StatusChecker.run(StatusChecker.java:33) at java.lang.Thread.run(Thread.java:724) [2014-03-09 14:15:36,805][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Closing river vendor_v1 [2014-03-09 14:15:36,807][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Closing Mongo client [2014-03-09 14:15:36,807][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Closing Mongo client [2014-03-09 14:15:36,808][INFO ][org.elasticsearch.river.mongodb.Slurper] river-mongodb slurper interrupted Exception in thread "elasticsearch[Penance][mongodb_river_indexer][T#1]" [2014-03-09 14:15:38,687][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Closing river review_v1 org.elasticsearch.ElasticsearchIllegalStateException: bulk process already closed [2014-03-09 14:15:38,687][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Closing Mongo client at org.elasticsearch.action.bulk.BulkProcessor.executeIfNeeded(BulkProcessor.java:255) at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:240) at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:234) at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:230) at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:216) at org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor.addBulkRequest(MongoDBRiverBulkProcessor.java:184)[2014-03-09 14:15:38,687][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Status thread interrupted java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at org.elasticsearch.river.mongodb.StatusChecker.run(StatusChecker.java:33) at java.lang.Thread.run(Thread.java:724)

at org.elasticsearch.river.mongodb.Indexer.updateBulkRequest(Indexer.java:246)
at org.elasticsearch.river.mongodb.Indexer.processBlockingQueue(Indexer.java:218)
at org.elasticsearch.river.mongodb.Indexer.run(Indexer.java:73)
at java.lang.Thread.run(Thread.java:724)

[2014-03-09 14:15:41,215][WARN ][org.elasticsearch.river.mongodb.Slurper] Exception while looping in cursor com.mongodb.MongoInterruptedException: A driver operation has been interrupted at com.mongodb.DBPortPool.get(DBPortPool.java:216) at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:380) at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:238) at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:216) at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:184) at com.mongodb.DBApiLayer$Result._advance(DBApiLayer.java:427) at com.mongodb.DBApiLayer$Result.hasNext(DBApiLayer.java:396) at com.mongodb.DBCursor._hasNext(DBCursor.java:464) at com.mongodb.DBCursor.hasNext(DBCursor.java:484) at org.elasticsearch.river.mongodb.Slurper.run(Slurper.java:123) at java.lang.Thread.run(Thread.java:724) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325) at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:414) at com.mongodb.util.SimplePool.permitAcquired(SimplePool.java:148) at com.mongodb.util.SimplePool.get(SimplePool.java:110) at com.mongodb.DBPortPool.get(DBPortPool.java:214) ... 10 more Mar 09, 2014 2:15:41 PM com.mongodb.DBApiLayer$Result killCursor WARNING: can't clean 1 cursor com.mongodb.MongoInterruptedException: A driver operation has been interrupted at com.mongodb.DBPortPool.get(DBPortPool.java:216) at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:380) at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:136) at com.mongodb.DBApiLayer.killCursors(DBApiLayer.java:175) at com.mongodb.DBApiLayer$Result.killCursor(DBApiLayer.java:486) at com.mongodb.DBApiLayer$Result.close(DBApiLayer.java:469) at com.mongodb.DBCursor.close(DBCursor.java:293) at org.elasticsearch.river.mongodb.Slurper.run(Slurper.java:142) at java.lang.Thread.run(Thread.java:724) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325) at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:414) at com.mongodb.util.SimplePool.permitAcquired(SimplePool.java:148) at com.mongodb.util.SimplePool.get(SimplePool.java:110) at com.mongodb.DBPortPool.get(DBPortPool.java:214) ... 8 more

[eshao]

I'm getting the same error as above. What is the recommended fix?

[dblado]

I still get this error -- using ES 1.1.0 and river plugin 2.0.0

I have two MongoDB collections that I'm loading through the river -- if I run them serially, there is no problem.

Here is my log file:

globogym:dblado elasticsearch-1.1.0$ ./bin/elasticsearch [2014-03-31 11:42:47,285][INFO ][node ] [Egghead] version[1.1.0], pid[1386], build[2181e11/2014-03-25T15:59:51Z] [2014-03-31 11:42:47,285][INFO ][node ] [Egghead] initializing ... [2014-03-31 11:42:47,339][INFO ][plugins ] [Egghead] loaded [mongodb-river, mapper-attachments, lang-python, index-termlist], sites [head, river-mongodb] [2014-03-31 11:42:49,487][INFO ][node ] [Egghead] initialized [2014-03-31 11:42:49,487][INFO ][node ] [Egghead] starting ... [2014-03-31 11:42:49,550][INFO ][transport ] [Egghead] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.10.150:9300]} [2014-03-31 11:42:52,574][INFO ][cluster.service ] [Egghead] new_master [Egghead][C-L3ihD6Rli3TGfu614EMw][globogym][inet[/192.168.10.150:9300]], reason: zen-disco-join (elected_as_master) [2014-03-31 11:42:52,591][INFO ][discovery ] [Egghead] elasticsearch/C-L3ihD6Rli3TGfu614EMw [2014-03-31 11:42:52,601][INFO ][http ] [Egghead] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.10.150:9200]} [2014-03-31 11:42:52,612][INFO ][gateway ] [Egghead] recovered [0] indices into cluster_state [2014-03-31 11:42:52,613][INFO ][node ] [Egghead] started

[2014-03-31 11:43:01,434][INFO ][cluster.metadata ] [Egghead] [vendop_v1] creating index, cause [api], shards [5]/[1], mappings [vendor, review]

[2014-03-31 11:43:10,548][INFO ][cluster.metadata ] [Egghead] [_river] creating index, cause [auto(index api)], shards [1]/[1], mappings [] [2014-03-31 11:43:10,598][INFO ][cluster.metadata ] [Egghead] [_river] update_mapping vendor_v1 [2014-03-31 11:43:10,630][INFO ][cluster.metadata ] [Egghead] [_river] update_mapping review_v1 [2014-03-31 11:43:10,642][INFO ][river.mongodb ] Parse river settings for review_v1 [2014-03-31 11:43:10,644][INFO ][river.mongodb ] Server: localhost - 27017 [2014-03-31 11:43:10,655][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Starting river review_v1 [2014-03-31 11:43:10,659][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB River Plugin - version[2.0.0] - hash[a0c23f1] - time[2014-02-23T20:40:05Z] [2014-03-31 11:43:10,659][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] starting mongodb stream. options: secondaryreadpreference [true], drop_collection [false], include_collection [], throttlesize [5000], gridfs [false], filter [null], db [vendop], collection [review], script [if ctx['document']['vendor']: ctx['_parent']=ctx['document']['vendor']], indexing to [vendop_v1]/[review] [2014-03-31 11:43:10,660][INFO ][cluster.metadata ] [Egghead] [_river] update_mapping review_v1 [2014-03-31 11:43:10,960][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB version - 2.4.9 [2014-03-31 11:43:10,972][INFO ][river.mongodb ] Parse river settings for vendor_v1 [2014-03-31 11:43:10,972][INFO ][river.mongodb ] Server: localhost - 27017 [2014-03-31 11:43:10,973][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Starting river vendor_v1 [2014-03-31 11:43:10,973][INFO ][cluster.metadata ] [Egghead] [_river] update_mapping review_v1 [2014-03-31 11:43:10,975][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB River Plugin - version[2.0.0] - hash[a0c23f1] - time[2014-02-23T20:40:05Z] [2014-03-31 11:43:10,975][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] starting mongodb stream. options: secondaryreadpreference [true], drop_collection [false], include_collection [], throttlesize [5000], gridfs [false], filter [null], db [vendop], collection [vendor], script [null], indexing to [vendop_v1]/[vendor] [2014-03-31 11:43:10,977][INFO ][cluster.metadata ] [Egghead] [_river] update_mapping vendor_v1 [2014-03-31 11:43:10,979][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB version - 2.4.9 [2014-03-31 11:43:10,984][INFO ][cluster.metadata ] [Egghead] [_river] update_mapping vendor_v1 [2014-03-31 11:43:11,024][INFO ][org.elasticsearch.river.mongodb.Slurper] MongoDBRiver is beginning initial import of vendop.review [2014-03-31 11:43:11,029][INFO ][org.elasticsearch.river.mongodb.Slurper] MongoDBRiver is beginning initial import of vendop.vendor [2014-03-31 11:43:15,614][INFO ][org.elasticsearch.river.mongodb.Slurper] Collection vendor - count: 5397307 [2014-03-31 11:43:15,615][INFO ][org.elasticsearch.river.mongodb.Slurper] Collection review - count: 2896 [2014-03-31 11:43:15,727][INFO ][cluster.metadata ] [Egghead] [vendop_v1] update_mapping vendor [2014-03-31 11:43:15,794][INFO ][cluster.metadata ] [Egghead] [vendop_v1] update_mapping vendor [2014-03-31 11:43:15,961][INFO ][org.elasticsearch.river.mongodb.Slurper] Number documents indexed: 2896 [2014-03-31 11:43:16,207][INFO ][cluster.metadata ] [Egghead] [vendop_v1] update_mapping vendor [2014-03-31 11:43:16,229][INFO ][cluster.metadata ] [Egghead] [vendop_v1] update_mapping vendor [2014-03-31 11:43:19,996][INFO ][cluster.metadata ] [Egghead] [vendop_v1] update_mapping review [2014-03-31 11:43:20,032][INFO ][org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor] Waiting for bulk queue to empty... [2014-03-31 11:43:20,173][ERROR][org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor] Bulk processor failed. failure in bulk execution: [6]: index [vendop_v1], type [review], id [52a8f64dc61e36756a976212], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2557ce03]] [10]: index [vendop_v1], type [review], id [52a8f84bc61e36756a976214], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2557ce03]] [16]: index [vendop_v1], type [review], id [52a8fbd5c61e36756931bcc3], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2557ce03]] [20]: index [vendop_v1], type [review], id [52a8fd1fc61e36756a97621a], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2557ce03]] [23]: index [vendop_v1], type [review], id [52a8ff59c61e36756a97621c], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2557ce03]] [32]: index [vendop_v1], type [review], id [52a90460c61e36756931bccc], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2557ce03]] [2014-03-31 11:43:20,185][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Closing river review_v1 [2014-03-31 11:43:20,185][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Closing Mongo client Exception in thread "elasticsearch[Egghead][mongodb_river_indexer][T#1]" [2014-03-31 11:43:20,195][INFO ][cluster.metadata ] [Egghead] [vendop_v1] update_mapping review org.elasticsearch.ElasticsearchIllegalStateException: bulk process already closed at org.elasticsearch.action.bulk.BulkProcessor.executeIfNeeded(BulkProcessor.java:255) at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:240) at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:234) at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:230) at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:216) at org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor.addBulkRequest(MongoDBRiverBulkProcessor.java:184) at org.elasticsearch.river.mongodb.Indexer.updateBulkRequest(Indexer.java:246) at org.elasticsearch.river.mongodb.Indexer.processBlockingQueue(Indexer.java:218) at org.elasticsearch.river.mongodb.Indexer.run(Indexer.java:73) at java.lang.Thread.run(Thread.java:724) [2014-03-31 11:43:20,201][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Status thread interrupted java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at org.elasticsearch.river.mongodb.StatusChecker.run(StatusChecker.java:33) at java.lang.Thread.run(Thread.java:724) [2014-03-31 11:43:20,975][WARN ][org.elasticsearch.river.mongodb.Slurper] Exception while looping in cursor com.mongodb.MongoInterruptedException: A driver operation has been interrupted at com.mongodb.DBPortPool.get(DBPortPool.java:216) at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:380) at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:238) at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:216) at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:184) at com.mongodb.DBApiLayer$Result._advance(DBApiLayer.java:427) at com.mongodb.DBApiLayer$Result.hasNext(DBApiLayer.java:396) at com.mongodb.DBCursor._hasNext(DBCursor.java:464) at com.mongodb.DBCursor.hasNext(DBCursor.java:484) at org.elasticsearch.river.mongodb.Slurper.run(Slurper.java:123) at java.lang.Thread.run(Thread.java:724) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325) at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:414) at com.mongodb.util.SimplePool.permitAcquired(SimplePool.java:148) at com.mongodb.util.SimplePool.get(SimplePool.java:110) at com.mongodb.DBPortPool.get(DBPortPool.java:214) ... 10 more Mar 31, 2014 11:43:20 AM com.mongodb.DBApiLayer$Result killCursor WARNING: can't clean 1 cursor com.mongodb.MongoInterruptedException: A driver operation has been interrupted at com.mongodb.DBPortPool.get(DBPortPool.java:216) at com.mongodb.DBTCPConnector$MyPort.get(DBTCPConnector.java:380) at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:136) at com.mongodb.DBApiLayer.killCursors(DBApiLayer.java:175) at com.mongodb.DBApiLayer$Result.killCursor(DBApiLayer.java:486) at com.mongodb.DBApiLayer$Result.close(DBApiLayer.java:469) at com.mongodb.DBCursor.close(DBCursor.java:293) at org.elasticsearch.river.mongodb.Slurper.run(Slurper.java:142) at java.lang.Thread.run(Thread.java:724) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1325) at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:414) at com.mongodb.util.SimplePool.permitAcquired(SimplePool.java:148) at com.mongodb.util.SimplePool.get(SimplePool.java:110) at com.mongodb.DBPortPool.get(DBPortPool.java:214) ... 8 more [2014-03-31 11:44:15,045][INFO ][river.mongodb ] Parse river settings for vendor_v1 [2014-03-31 11:44:15,045][INFO ][river.mongodb ] Server: localhost - 27017 [2014-03-31 11:44:15,168][INFO ][river.mongodb ] Parse river settings for review_v1 [2014-03-31 11:44:15,168][INFO ][river.mongodb ] Server: localhost - 27017

[richardwilly98]

@dblado Do you see any trace in ES log file like Waiting for bulk queue to empty...? There is a check in the river to wait if bulk queue is almost full

[nickchang]

@richardwilly98 I get same error: [2014-04-08 22:29:42,367][INFO ][org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor] Waiting for bulk queue to empty... [2014-04-08 22:29:43,218][ERROR][org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor] Bulk processor failed. failure in bulk execution: [3]: index [reporting], type [reporting], id [51f7212943e350ed85bd75b8], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@ca4450f]] [7]: index [reporting], type [reporting], id [51f7216543e350ed85bd75bc], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@ca4450f]] [12]: index [reporting], type [reporting], id [51f7223643e350ed85bd75c1], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@ca4450f]] [19]: index [reporting], type [reporting], id [51f724a243e350ed85bd75c9], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@ca4450f]]

How to solve this problem?? My ES Version is 1.1.0., River Version is 2.0

[dblado]

@richardwilly98 here is the log -- nothing about waiting for bulk queue

[2014-04-13 14:43:11,622][INFO ][cluster.metadata ] [Ego the Living Planet] [_river] update_mapping vendor_v2 [2014-04-13 14:43:11,624][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] Initializing river : [vendor_v2] [2014-04-13 14:43:11,625][INFO ][river.mongodb ] Parse river settings for vendor_v2 [2014-04-13 14:43:11,625][TRACE][river.mongodb ] mongoServersSettings: [{port=27017, host=localhost}] [2014-04-13 14:43:11,625][INFO ][river.mongodb ] Server: localhost - 27017 [2014-04-13 14:43:11,625][TRACE][river.mongodb ] mongoOptionsSettings: {secondary_read_preference=true} [2014-04-13 14:43:11,625][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Starting river vendor_v2 [2014-04-13 14:43:11,625][DEBUG][river.mongodb.util ] setRiverStatus called with vendor_v2 - RUNNING [2014-04-13 14:43:11,626][DEBUG][org.elasticsearch.river.mongodb.MongoDBRiver] Using mongodb server(s): host [localhost], port [27017] [2014-04-13 14:43:11,627][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB River Plugin - version[2.0.0] - hash[a0c23f1] - time[2014-02-23T20:40:05Z] [2014-04-13 14:43:11,627][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] starting mongodb stream. options: secondaryreadpreference [true], drop_collection [false], include_collection [], throttlesize [5000], gridfs [false], filter [null], db [vendop], collection [vendor], script [null], indexing to [vendop_v2]/[vendor] [2014-04-13 14:43:11,627][INFO ][cluster.metadata ] [Ego the Living Planet] [_river] update_mapping vendor_v2 [2014-04-13 14:43:11,627][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] MongoAdminUser: - authenticated: false [2014-04-13 14:43:11,627][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] Found admin database [2014-04-13 14:43:11,628][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] About to execute: { "serverStatus" : 1 , "asserts" : 0 , "backgroundFlushing" : 0 , "connections" : 0 , "cursors" : 0 , "dur" : 0 , "extra_info" : 0 , "globalLock" : 0 , "indexCounters" : 0 , "locks" : 0 , "metrics" : 0 , "network" : 0 , "opcounters" : 0 , "opcountersRepl" : 0 , "recordStats" : 0 , "repl" : 0} [2014-04-13 14:43:11,631][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] Command executed return : { "serverUsed" : "globogym.local/169.254.197.120:27017" , "host" : "globogym" , "version" : "2.4.9" , "process" : "mongod" , "pid" : 4949 , "uptime" : 9979.0 , "uptimeMillis" : 9978929 , "uptimeEstimate" : 9166.0 , "localTime" : { "$date" : "2014-04-13T21:43:11.630Z"} , "ok" : 1.0} [2014-04-13 14:43:11,631][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB version - 2.4.9 [2014-04-13 14:43:11,631][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] serverStatus: { "serverUsed" : "globogym.local/169.254.197.120:27017" , "host" : "globogym" , "version" : "2.4.9" , "process" : "mongod" , "pid" : 4949 , "uptime" : 9979.0 , "uptimeMillis" : 9978929 , "uptimeEstimate" : 9166.0 , "localTime" : { "$date" : "2014-04-13T21:43:11.630Z"} , "ok" : 1.0} [2014-04-13 14:43:11,631][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] process: mongod [2014-04-13 14:43:11,631][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] Not mongos [2014-04-13 14:43:11,634][INFO ][cluster.metadata ] [Ego the Living Planet] [_river] update_mapping review_v2 [2014-04-13 14:43:11,638][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] Initializing river : [review_v2] [2014-04-13 14:43:11,638][INFO ][river.mongodb ] Parse river settings for review_v2 [2014-04-13 14:43:11,638][TRACE][river.mongodb ] mongoServersSettings: [{port=27017, host=localhost}] [2014-04-13 14:43:11,638][INFO ][river.mongodb ] Server: localhost - 27017 [2014-04-13 14:43:11,638][TRACE][river.mongodb ] mongoOptionsSettings: {secondary_read_preference=true} [2014-04-13 14:43:11,638][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Starting river review_v2 [2014-04-13 14:43:11,639][DEBUG][river.mongodb.util ] setRiverStatus called with review_v2 - RUNNING [2014-04-13 14:43:11,639][INFO ][cluster.metadata ] [Ego the Living Planet] [_river] update_mapping vendor_v2 [2014-04-13 14:43:11,640][DEBUG][org.elasticsearch.river.mongodb.MongoDBRiver] Using mongodb server(s): host [localhost], port [27017] [2014-04-13 14:43:11,641][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB River Plugin - version[2.0.0] - hash[a0c23f1] - time[2014-02-23T20:40:05Z] [2014-04-13 14:43:11,641][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] starting mongodb stream. options: secondaryreadpreference [true], drop_collection [false], include_collection [], throttlesize [5000], gridfs [false], filter [null], db [vendop], collection [review], script [if ctx['document']['vendor']: ctx['_parent']=ctx['document']['vendor']], indexing to [vendop_v2]/[review] [2014-04-13 14:43:11,642][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] MongoAdminUser: - authenticated: false [2014-04-13 14:43:11,642][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] Found admin database [2014-04-13 14:43:11,642][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] About to execute: { "serverStatus" : 1 , "asserts" : 0 , "backgroundFlushing" : 0 , "connections" : 0 , "cursors" : 0 , "dur" : 0 , "extra_info" : 0 , "globalLock" : 0 , "indexCounters" : 0 , "locks" : 0 , "metrics" : 0 , "network" : 0 , "opcounters" : 0 , "opcountersRepl" : 0 , "recordStats" : 0 , "repl" : 0} [2014-04-13 14:43:11,644][INFO ][org.elasticsearch.river.mongodb.Slurper] MongoDBRiver is beginning initial import of vendop.vendor [2014-04-13 14:43:11,644][INFO ][cluster.metadata ] [Ego the Living Planet] [_river] update_mapping review_v2 [2014-04-13 14:43:11,647][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] Command executed return : { "serverUsed" : "globogym.local/169.254.197.120:27017" , "host" : "globogym" , "version" : "2.4.9" , "process" : "mongod" , "pid" : 4949 , "uptime" : 9979.0 , "uptimeMillis" : 9978945 , "uptimeEstimate" : 9166.0 , "localTime" : { "$date" : "2014-04-13T21:43:11.646Z"} , "ok" : 1.0} [2014-04-13 14:43:11,647][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB version - 2.4.9 [2014-04-13 14:43:11,647][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] serverStatus: { "serverUsed" : "globogym.local/169.254.197.120:27017" , "host" : "globogym" , "version" : "2.4.9" , "process" : "mongod" , "pid" : 4949 , "uptime" : 9979.0 , "uptimeMillis" : 9978945 , "uptimeEstimate" : 9166.0 , "localTime" : { "$date" : "2014-04-13T21:43:11.646Z"} , "ok" : 1.0} [2014-04-13 14:43:11,647][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] process: mongod [2014-04-13 14:43:11,647][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] Not mongos [2014-04-13 14:43:11,652][INFO ][cluster.metadata ] [Ego the Living Planet] [_river] update_mapping review_v2 [2014-04-13 14:43:11,658][INFO ][org.elasticsearch.river.mongodb.Slurper] MongoDBRiver is beginning initial import of vendop.review [2014-04-13 14:43:11,662][INFO ][org.elasticsearch.river.mongodb.Slurper] Collection vendor - count: 5397844 [2014-04-13 14:43:11,674][INFO ][org.elasticsearch.river.mongodb.Slurper] Collection review - count: 15953 [2014-04-13 14:43:11,708][INFO ][cluster.metadata ] [Ego the Living Planet] [vendop_v2] update_mapping vendor [2014-04-13 14:43:11,714][INFO ][cluster.metadata ] [Ego the Living Planet] [vendop_v2] update_mapping review [2014-04-13 14:43:11,819][INFO ][cluster.metadata ] [Ego the Living Planet] [vendop_v2] update_mapping vendor [2014-04-13 14:43:11,881][INFO ][cluster.metadata ] [Ego the Living Planet] [vendop_v2] update_mapping review [2014-04-13 14:43:12,488][ERROR][org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor] Bulk processor failed. failure in bulk execution: [0]: index [vendop_v2], type [review], id [530fc9f3c61e364335010b1e], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [1]: index [vendop_v2], type [review], id [530fca32c61e364335010b1f], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [2]: index [vendop_v2], type [review], id [530fcacbc61e3643338ce5e3], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [3]: index [vendop_v2], type [review], id [530fcb2dc61e364334d0b5fa], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [4]: index [vendop_v2], type [review], id [530fcb2dc61e3643338ce5e4], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [6]: index [vendop_v2], type [review], id [530fcc01c61e3643338ce5e5], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [7]: index [vendop_v2], type [review], id [530fcc1ec61e364334d0b5fb], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [9]: index [vendop_v2], type [review], id [530fccadc61e364334d0b5fc], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [10]: index [vendop_v2], type [review], id [530fccb9c61e364334d0b5fd], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [11]: index [vendop_v2], type [review], id [530fccddc61e36433678d9ab], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [12]: index [vendop_v2], type [review], id [530fcd67c61e364334d0b5ff], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [13]: index [vendop_v2], type [review], id [530fcdccc61e364330b3aa65], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [14]: index [vendop_v2], type [review], id [530fce06c61e364334d0b600], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [15]: index [vendop_v2], type [review], id [530fce2dc61e36433678d9ac], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [16]: index [vendop_v2], type [review], id [530fcf0ec61e36433678d9ad], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [18]: index [vendop_v2], type [review], id [530fd045c61e364334d0b602], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [19]: index [vendop_v2], type [review], id [530fd08dc61e36433678d9af], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [20]: index [vendop_v2], type [review], id [530fd177c61e364332fb3d1a], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [23]: index [vendop_v2], type [review], id [530fd27dc61e364334d0b603], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [24]: index [vendop_v2], type [review], id [530fd322c61e3643338ce5e8], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [27]: index [vendop_v2], type [review], id [530fd40bc61e364334d0b606], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [28]: index [vendop_v2], type [review], id [530fd47ac61e364334d0b607], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [29]: index [vendop_v2], type [review], id [530fd517c61e36433678d9b1], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [30]: index [vendop_v2], type [review], id [530fd53ac61e36433678d9b2], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [31]: index [vendop_v2], type [review], id [530fd567c61e364334d0b608], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [32]: index [vendop_v2], type [review], id [530fd5f4c61e36433678d9b3], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [33]: index [vendop_v2], type [review], id [530fd69ec61e36433678d9b4], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [35]: index [vendop_v2], type [review], id [530fd740c61e364334d0b609], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [37]: index [vendop_v2], type [review], id [530fd8dcc61e364334d0b60c], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [38]: index [vendop_v2], type [review], id [530fd933c61e364334d0b60d], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [39]: index [vendop_v2], type [review], id [530fd9c4c61e3643338ce5e9], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [40]: index [vendop_v2], type [review], id [530fd9fec61e364335010b23], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [41]: index [vendop_v2], type [review], id [530fdad7c61e36433678d9b6], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [42]: index [vendop_v2], type [review], id [530fdb06c61e364334d0b60f], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [43]: index [vendop_v2], type [review], id [530fdb79c61e364334d0b610], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [44]: index [vendop_v2], type [review], id [530fdcf5c61e364334d0b611], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [45]: index [vendop_v2], type [review], id [530fdd1bc61e36433678d9b7], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [46]: index [vendop_v2], type [review], id [530fde7fc61e3643316e6924], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [49]: index [vendop_v2], type [review], id [530fe022c61e364334d0b614], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [51]: index [vendop_v2], type [review], id [530fe155c61e36433678d9b8], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [52]: index [vendop_v2], type [review], id [530fe158c61e364335010b24], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [53]: index [vendop_v2], type [review], id [530fe226c61e3643338ce5ea], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [54]: index [vendop_v2], type [review], id [530fe24cc61e36433678d9b9], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [55]: index [vendop_v2], type [review], id [530fe39dc61e36433678d9ba], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [56]: index [vendop_v2], type [review], id [530fe44dc61e364334d0b616], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [57]: index [vendop_v2], type [review], id [530fe46ac61e364334d0b617], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [58]: index [vendop_v2], type [review], id [530fe521c61e364334d0b618], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [59]: index [vendop_v2], type [review], id [530fe578c61e36433678d9bb], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [60]: index [vendop_v2], type [review], id [530fe630c61e364334d0b619], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [61]: index [vendop_v2], type [review], id [530fe69cc61e36433678d9bc], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [62]: index [vendop_v2], type [review], id [530fe6a2c61e364334d0b61a], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [64]: index [vendop_v2], type [review], id [530fe74cc61e364334d0b61c], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [65]: index [vendop_v2], type [review], id [530fe752c61e364334d0b61d], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [66]: index [vendop_v2], type [review], id [530fe7d1c61e364334d0b61e], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [67]: index [vendop_v2], type [review], id [530fe7d2c61e364334d0b61f], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [68]: index [vendop_v2], type [review], id [530fe8dcc61e36433678d9bd], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [69]: index [vendop_v2], type [review], id [530fe8e8c61e364334d0b620], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [70]: index [vendop_v2], type [review], id [530fe8eac61e364334d0b621], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [71]: index [vendop_v2], type [review], id [530fe922c61e364334d0b622], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [72]: index [vendop_v2], type [review], id [530fe97bc61e364334d0b623], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [73]: index [vendop_v2], type [review], id [530fe982c61e364334d0b624], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [74]: index [vendop_v2], type [review], id [530fe9efc61e36433678d9be], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [75]: index [vendop_v2], type [review], id [530fea28c61e36433678d9bf], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [76]: index [vendop_v2], type [review], id [530fea33c61e364334d0b625], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [77]: index [vendop_v2], type [review], id [530fea6cc61e36433678d9c0], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [78]: index [vendop_v2], type [review], id [530fea9ec61e364334d0b626], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [79]: index [vendop_v2], type [review], id [530feb29c61e3643338ce5eb], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [80]: index [vendop_v2], type [review], id [530feb65c61e36433678d9c1], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [81]: index [vendop_v2], type [review], id [530feb6ec61e364334d0b627], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [82]: index [vendop_v2], type [review], id [530feb81c61e364334d0b628], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [83]: index [vendop_v2], type [review], id [530febe1c61e364334d0b629], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [85]: index [vendop_v2], type [review], id [530fec29c61e36433678d9c3], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [86]: index [vendop_v2], type [review], id [530fec7fc61e364334d0b62a], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [87]: index [vendop_v2], type [review], id [530feccac61e364334d0b62b], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [89]: index [vendop_v2], type [review], id [530fed71c61e36433678d9c4], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [90]: index [vendop_v2], type [review], id [530fedfdc61e364334d0b62e], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [91]: index [vendop_v2], type [review], id [530fee16c61e364334d0b62f], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [92]: index [vendop_v2], type [review], id [530feeb7c61e364334d0b630], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [93]: index [vendop_v2], type [review], id [530feecbc61e364334d0b631], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [94]: index [vendop_v2], type [review], id [530feef1c61e364334d0b632], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [95]: index [vendop_v2], type [review], id [530fef6ec61e364334d0b633], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [96]: index [vendop_v2], type [review], id [530fefcdc61e364334d0b634], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [97]: index [vendop_v2], type [review], id [530ff06ec61e36433678d9c6], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [99]: index [vendop_v2], type [review], id [530ff30cc61e364334d0b636], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [101]: index [vendop_v2], type [review], id [530ff4d6c61e364334d0b638], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [102]: index [vendop_v2], type [review], id [530ff525c61e364334d0b639], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [103]: index [vendop_v2], type [review], id [530ff8a2c61e36433678d9c7], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [104]: index [vendop_v2], type [review], id [530ff8c3c61e364334d0b63a], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [105]: index [vendop_v2], type [review], id [530ff931c61e364334d0b63b], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [106]: index [vendop_v2], type [review], id [530ff9b7c61e364334d0b63c], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [107]: index [vendop_v2], type [review], id [530ffa6bc61e364334d0b63d], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [108]: index [vendop_v2], type [review], id [530ffa75c61e364334d0b63e], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [109]: index [vendop_v2], type [review], id [530ffb99c61e364334d0b63f], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@434e9bb5]] [110]: index [vendop_v2], type [review], id [530ffc13c61e364334d0b640], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [111]: index [vendop_v2], type [review], id [530ffcbcc61e364334d0b641], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [112]: index [vendop_v2], type [review], id [530ffd2cc61e36433678d9c8], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [115]: index [vendop_v2], type [review], id [530ffd66c61e364334d0b644], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [116]: index [vendop_v2], type [review], id [530ffdedc61e364334d0b645], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [117]: index [vendop_v2], type [review], id [530ffe76c61e364334d0b646], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [119]: index [vendop_v2], type [review], id [530fffa6c61e364334d0b649], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [120]: index [vendop_v2], type [review], id [53100053c61e364334d0b64a], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [121]: index [vendop_v2], type [review], id [53100054c61e364334d0b64b], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [122]: index [vendop_v2], type [review], id [53100059c61e36433678d9c9], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [123]: index [vendop_v2], type [review], id [53100078c61e36433678d9ca], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@5dd0aa98]] [125]: index [vendop_v2], type [review], id [531001e1c61e36433678d9cb], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@3f7e227b]] [126]: index [vendop_v2], type [review], id [53100254c61e364334d0b64d], message [EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@2e701146]] [2014-04-13 14:43:12,490][DEBUG][river.mongodb.util ] setRiverStatus called with review_v2 - IMPORT_FAILED [2014-04-13 14:43:12,505][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Closing river review_v2 [2014-04-13 14:43:12,505][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Closing Mongo client Exception in thread "elasticsearch[Ego the Living Planet][mongodb_river_indexer][T#1]" org.elasticsearch.ElasticsearchIllegalStateException: bulk process already closed at org.elasticsearch.action.bulk.BulkProcessor.executeIfNeeded(BulkProcessor.java:255) at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:240) at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:234) at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:230) at org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:216) at org.elasticsearch.river.mongodb.MongoDBRiverBulkProcessor.addBulkRequest(MongoDBRiverBulkProcessor.java:184) at org.elasticsearch.river.mongodb.Indexer.updateBulkRequest(Indexer.java:246) at org.elasticsearch.river.mongodb.Indexer.processBlockingQueue(Indexer.java:218) at org.elasticsearch.river.mongodb.Indexer.run(Indexer.java:73) at java.lang.Thread.run(Thread.java:724) [2014-04-13 14:43:12,505][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Status thread interrupted java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at org.elasticsearch.river.mongodb.StatusChecker.run(StatusChecker.java:33) at java.lang.Thread.run(Thread.java:724) [2014-04-13 14:43:12,505][INFO ][org.elasticsearch.river.mongodb.Slurper] river-mongodb slurper interrupted [

[sampurn]

is there a way to skip certain records which it is unable to map due to some exception while the river updates the mapping ?? i get mapping exception on some inconsistent data