elastic search river mongodb not indexing the documents

rajeeshv commented 9 years ago

I am trying to setup elastic search river mongodb. The versions that I am using MongoDB - 2.6.5 Elastic search - 1.4.0 Elastic search river plugin - 2.0.5 ES mapper attachement - 2.4.2

I setup mongodb using this http://docs.mongodb.org/manual/tutorial/convert-standalone-to-replica-set/ Setup elastic search river using this https://github.com/richardwilly98/elasticsearch-river-mongodb/wiki

I am not getting any document indexed.

Immediately after startup I am getting the log like this

[2014-12-10 15:44:12,987][INFO ][http ] [Eddie Brock] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.3.0.53:9200]} [2014-12-10 15:44:12,987][INFO ][node ] [Eddie Brock] started [2014-12-10 15:44:13,744][INFO ][gateway ] [Eddie Brock] recovered [9] indices into cluster_state [2014-12-10 15:44:16,422][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] Initializing river : [mongodb] [2014-12-10 15:44:16,423][TRACE][river.mongodb ] Parse river settings for mongodb [2014-12-10 15:44:16,443][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Starting river mongodb [2014-12-10 15:44:16,448][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB River Plugin - version[2.0.5-SNAPSHOT] - hash[${build] - time[2014-11-28T12:36:31Z] [2014-12-10 15:44:16,449][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] starting mongodb stream. options: secondaryreadpreference [false], drop_collection [false], include_collection [], throttlesize [5000], gridfs [false], filter [null], db [testmongo], collection [person], script [null], indexing to [mongoindex]/[person] [2014-12-10 15:44:16,449][INFO ][river.mongodb.util ] setRiverStatus called with mongodb - RUNNING [2014-12-10 15:44:16,517][DEBUG][org.elasticsearch.river.mongodb.MongoDBRiver] Using mongodb server(s): host [localhost], port [27017]

dharshanr commented 9 years ago

Hi Rajeev,

Are you on a single proc machine? Then its possible that the thread pool does not have capacity. See if you can increase the "listerner threadpool".

threadpool: listener: type: fixedsize: 5

rajeeshv commented 9 years ago

My machine is 2.60GHz × 2 (dual core). River worked when I downgrade Elastic search to 1.3.5

sokser commented 9 years ago

Does updating and deleting data from mongo get synced into your elasticsearch? I get new items added but nothing updated or deleted, even if the logs indicate that the action has been read from the oplogs.

rajeeshv commented 9 years ago

I checked both update and delete with 1.3.5, it working.

ankon commented 9 years ago

Could you try with current master? This could be a duplicate of #406/#407.

What could also help is this:

Are you using multiple rivers, or just one?
Can you provide a full output of jstack <PID-of-elasticsearch>

rajeeshv commented 9 years ago

I had tried with current master of elasticsearch river and mapper attachment as on 10 Dec. Elastic search used is 1.4, not master. I only have one river I tried jstack, getting an error, might need to update my java for solving that.

ankon commented 9 years ago

Sorry for being unclear: current master of the elasticsearch-river-mongodb, i.e. with #407 merged (which happened yesterday). This should be compatible with elasticsearch 1.4.x.

Regarding the jstack error: the most common one is simply a version mismatch, make sure to use the jstack you find inside the bin directory of the same java you use for running elasticsearch. A second common issue occurs when elasticsearch is running under a different user, so you should run jstack under the user that is used for elasticsearch.

Please do post the error if those steps don't help, while we don't have much time to debug it, it is better used when we don't have to guess what's going on :)

tmatei commented 9 years ago

@ankon Got any ideas why I can't compile this on my mac? I'm running java 1.8.0_25

mvn -Dmaven.test.skip=true package
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building elasticsearch-river-mongodb 2.0.5-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-enforcer-plugin:1.0:enforce (enforce-maven) @ elasticsearch-river-mongodb ---
[INFO]
[INFO] --- buildnumber-maven-plugin:1.2:create (default) @ elasticsearch-river-mongodb ---
[INFO] Checking for local modifications: skipped.
[INFO] Updating project files from SCM: skipped.
[INFO] Executing: /bin/sh -c cd /Users/tudor/elasticsearch-river-mongodb && git rev-parse --verify HEAD
[INFO] Working directory: /Users/tudor/elasticsearch-river-mongodb
[INFO] Storing buildNumber: 1e2ff4d6938866921f0c986f3aa7786fd2584164 at timestamp: 1418749192274
[INFO] Executing: /bin/sh -c cd /Users/tudor/elasticsearch-river-mongodb && git rev-parse --verify HEAD
[INFO] Working directory: /Users/tudor/elasticsearch-river-mongodb
[INFO] Storing buildScmBranch: UNKNOWN
[INFO]
[INFO] --- build-helper-maven-plugin:1.5:add-source (add-source) @ elasticsearch-river-mongodb ---
[INFO] Source directory: /Users/tudor/elasticsearch-river-mongodb/src/main/groovy added.
[INFO]
[INFO] --- build-helper-maven-plugin:1.5:add-resource (add-resource) @ elasticsearch-river-mongodb ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ elasticsearch-river-mongodb ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 1 resource to /Users/tudor/elasticsearch-river-mongodb/target/config/scripts
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ elasticsearch-river-mongodb ---
[INFO] Changes detected - recompiling the module!
[INFO] Using Groovy-Eclipse compiler to compile both Java and Groovy files
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/util/MongoDBHelper.java:[58,6] 1. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/util/MongoDBHelper.java (at line 58)
    try (InputStream stream = file.getInputStream()) {
         ^^^^^^^^^^^
The resource type InputStream does not implement java.lang.AutoCloseable

[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/util/MongoDBHelper.java:[59,17] 2. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/util/MongoDBHelper.java (at line 59)
    while ((nRead = stream.read(data, 0, data.length)) != -1) {
                    ^^^^^^
The type InputStream is not visible

[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoDBRiver.java:[29,8] 3. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoDBRiver.java (at line 29)
    import java.util.concurrent.LinkedTransferQueue;
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The import java.util.concurrent.LinkedTransferQueue cannot be resolved

[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoDBRiver.java:[141,77] 4. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoDBRiver.java (at line 141)
    BlockingQueue<QueueEntry> stream = definition.getThrottleSize() == -1 ? new LinkedTransferQueue<QueueEntry>()
                                                                                ^^^^^^^^^^^^^^^^^^^
LinkedTransferQueue cannot be resolved to a type

[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java:[120,6] 5. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java (at line 120)
    try (DBCursor cursor = getConfigDb().getCollection("shards").find()) {
         ^^^^^^^^
The resource type DBCursor does not implement java.lang.AutoCloseable

[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java:[121,8] 6. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java (at line 121)
    while (cursor.hasNext()) {
           ^^^^^^
The type DBCursor is not visible

[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java:[122,17] 7. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java (at line 122)
    DBObject item = cursor.next();
                    ^^^^^^
The type DBCursor is not visible

[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java:[164,6] 8. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java (at line 164)
    try (DBCursor cursor = oplogCollection.find().sort(new BasicDBObject("$natural", -1)).limit(1)) {
         ^^^^^^^^
The resource type DBCursor does not implement java.lang.AutoCloseable

[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java:[165,21] 9. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java (at line 165)
    return Timestamp.on(cursor.next());
                        ^^^^^^
The type DBCursor is not visible

[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/OplogSlurper.java:[155,6] 10. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/OplogSlurper.java (at line 155)
    try (DBCursor cursor = oplogCollection.find().sort(new BasicDBObject(MongoDBRiver.INSERTION_ORDER_KEY, -1)).limit(1)) {
         ^^^^^^^^
The resource type DBCursor does not implement java.lang.AutoCloseable

[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/OplogSlurper.java:[156,21] 11. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/OplogSlurper.java (at line 156)
    return Timestamp.on(cursor.next());
                        ^^^^^^
The type DBCursor is not visible

[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/OplogSlurper.java:[511,6] 12. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/OplogSlurper.java (at line 511)
    try (DBCursor cursor = slurpedCollection.find(update, findKeys)) {
         ^^^^^^^^
The resource type DBCursor does not implement java.lang.AutoCloseable

[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/OplogSlurper.java:[512,22] 13. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/OplogSlurper.java (at line 512)
    for (DBObject item : cursor) {
                         ^^^^^^
Can only iterate over an array or an instance of java.lang.Iterable

[ERROR] Found 13 errors and 0 warnings.
[INFO] 14 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.018 s
[INFO] Finished at: 2014-12-16T08:59:54-08:00
[INFO] Final Memory: 11M/81M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project elasticsearch-river-mongodb: Compilation failure: Compilation failure:
[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/util/MongoDBHelper.java:[58,6] 1. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/util/MongoDBHelper.java (at line 58)
[ERROR] try (InputStream stream = file.getInputStream()) {
[ERROR] ^^^^^^^^^^^
[ERROR] The resource type InputStream does not implement java.lang.AutoCloseable
[ERROR]
[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/util/MongoDBHelper.java:[59,17] 2. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/util/MongoDBHelper.java (at line 59)
[ERROR] while ((nRead = stream.read(data, 0, data.length)) != -1) {
[ERROR] ^^^^^^
[ERROR] The type InputStream is not visible
[ERROR]
[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoDBRiver.java:[29,8] 3. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoDBRiver.java (at line 29)
[ERROR] import java.util.concurrent.LinkedTransferQueue;
[ERROR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ERROR] The import java.util.concurrent.LinkedTransferQueue cannot be resolved
[ERROR]
[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoDBRiver.java:[141,77] 4. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoDBRiver.java (at line 141)
[ERROR] BlockingQueue<QueueEntry> stream = definition.getThrottleSize() == -1 ? new LinkedTransferQueue<QueueEntry>()
[ERROR] ^^^^^^^^^^^^^^^^^^^
[ERROR] LinkedTransferQueue cannot be resolved to a type
[ERROR]
[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java:[120,6] 5. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java (at line 120)
[ERROR] try (DBCursor cursor = getConfigDb().getCollection("shards").find()) {
[ERROR] ^^^^^^^^
[ERROR] The resource type DBCursor does not implement java.lang.AutoCloseable
[ERROR]
[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java:[121,8] 6. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java (at line 121)
[ERROR] while (cursor.hasNext()) {
[ERROR] ^^^^^^
[ERROR] The type DBCursor is not visible
[ERROR]
[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java:[122,17] 7. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java (at line 122)
[ERROR] DBObject item = cursor.next();
[ERROR] ^^^^^^
[ERROR] The type DBCursor is not visible
[ERROR]
[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java:[164,6] 8. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java (at line 164)
[ERROR] try (DBCursor cursor = oplogCollection.find().sort(new BasicDBObject("$natural", -1)).limit(1)) {
[ERROR] ^^^^^^^^
[ERROR] The resource type DBCursor does not implement java.lang.AutoCloseable
[ERROR]
[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java:[165,21] 9. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/MongoConfigProvider.java (at line 165)
[ERROR] return Timestamp.on(cursor.next());
[ERROR] ^^^^^^
[ERROR] The type DBCursor is not visible
[ERROR]
[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/OplogSlurper.java:[155,6] 10. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/OplogSlurper.java (at line 155)
[ERROR] try (DBCursor cursor = oplogCollection.find().sort(new BasicDBObject(MongoDBRiver.INSERTION_ORDER_KEY, -1)).limit(1)) {
[ERROR] ^^^^^^^^
[ERROR] The resource type DBCursor does not implement java.lang.AutoCloseable
[ERROR]
[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/OplogSlurper.java:[156,21] 11. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/OplogSlurper.java (at line 156)
[ERROR] return Timestamp.on(cursor.next());
[ERROR] ^^^^^^
[ERROR] The type DBCursor is not visible
[ERROR]
[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/OplogSlurper.java:[511,6] 12. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/OplogSlurper.java (at line 511)
[ERROR] try (DBCursor cursor = slurpedCollection.find(update, findKeys)) {
[ERROR] ^^^^^^^^
[ERROR] The resource type DBCursor does not implement java.lang.AutoCloseable
[ERROR]
[ERROR] /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/OplogSlurper.java:[512,22] 13. ERROR in /Users/tudor/elasticsearch-river-mongodb/src/main/java/org/elasticsearch/river/mongodb/OplogSlurper.java (at line 512)
[ERROR] for (DBObject item : cursor) {
[ERROR] ^^^^^^
[ERROR] Can only iterate over an array or an instance of java.lang.Iterable
[ERROR]
[ERROR] Found 13 errors and 0 warnings.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

ankon commented 9 years ago

@tmatei doesn't look like Java 8, can you run with mvn -X and see what it logs as java home/version.

tmatei commented 9 years ago

@ankon Thank you I was using the wrong version of java for mv, weird. So I was able to get the plugin to work on my mac, but not on my CentOS dev (both are running 1.4.2 with the latest master plugin) I printed out the jstack of the process id. I hope this helps.The problem persists with ES not responding

sudo jstack -F 5430
Attaching to process ID 5430, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.25-b02
Deadlock Detection:

No deadlocks found.

Thread 5548: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=184, line=737 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.take() @bci=5, line=1265 (Interpreted frame)
 - org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take() @bci=4, line=162 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5547: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=184, line=737 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.take() @bci=5, line=1265 (Interpreted frame)
 - org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take() @bci=4, line=162 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5491: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=177, line=734 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1273 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5490: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=177, line=734 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1273 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5489: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=177, line=734 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1273 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5488: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=177, line=734 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1273 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5487: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=184, line=737 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.take() @bci=5, line=1265 (Interpreted frame)
 - org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take() @bci=4, line=162 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5486: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=184, line=737 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.take() @bci=5, line=1265 (Interpreted frame)
 - org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take() @bci=4, line=162 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5485: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=184, line=737 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.take() @bci=5, line=1265 (Interpreted frame)
 - org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take() @bci=4, line=162 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5484: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=184, line=737 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.take() @bci=5, line=1265 (Interpreted frame)
 - org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take() @bci=4, line=162 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5481: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=184, line=737 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.take() @bci=5, line=1265 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5480: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=184, line=737 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.take() @bci=5, line=1265 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5479: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=184, line=737 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.take() @bci=5, line=1265 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5478: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=184, line=737 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.take() @bci=5, line=1265 (Interpreted frame)
 - org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take() @bci=4, line=162 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5477: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int) @bci=72, line=997 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int) @bci=24, line=1304 (Compiled frame)
 - org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get() @bci=2, line=274 (Interpreted frame)
 - org.elasticsearch.common.util.concurrent.BaseFuture.get() @bci=4, line=113 (Interpreted frame)
 - org.elasticsearch.action.support.AdapterActionFuture.actionGet() @bci=1, line=45 (Interpreted frame)
 - org.elasticsearch.action.ActionRequestBuilder.get() @bci=4, line=73 (Interpreted frame)
 - org.elasticsearch.river.mongodb.MongoDBRiver.start() @bci=442, line=178 (Interpreted frame)
 - org.elasticsearch.river.RiversService.createRiver(org.elasticsearch.river.RiverName, java.util.Map) @bci=204, line=148 (Interpreted frame)
 - org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(org.elasticsearch.action.get.GetResponse) @bci=48, line=275 (Interpreted frame)
 - org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(java.lang.Object) @bci=5, line=269 (Interpreted frame)
 - org.elasticsearch.action.support.TransportAction$ThreadedActionListener$1.run() @bci=11, line=113 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5476: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=184, line=737 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.take() @bci=5, line=1265 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5475: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=184, line=737 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.take() @bci=5, line=1265 (Interpreted frame)
 - org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take() @bci=4, line=162 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5474: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=184, line=737 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.take() @bci=5, line=1265 (Interpreted frame)
 - org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take() @bci=4, line=162 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5473: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=160, line=460 (Interpreted frame)
 - java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=362 (Interpreted frame)
 - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=941 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5432: (state = BLOCKED)

Thread 5471: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int) @bci=72, line=997 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int) @bci=24, line=1304 (Compiled frame)
 - java.util.concurrent.CountDownLatch.await() @bci=5, line=231 (Interpreted frame)
 - org.elasticsearch.bootstrap.Bootstrap$3.run() @bci=3, line=225 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5470: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select() @bci=2, line=101 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioServerBoss.select(java.nio.channels.Selector) @bci=1, line=163 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioServerBoss.run() @bci=1, line=42 (Interpreted frame)
 - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5469: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int) @bci=72, line=997 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int) @bci=24, line=1304 (Compiled frame)
 - org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get() @bci=2, line=274 (Interpreted frame)
 - org.elasticsearch.common.util.concurrent.BaseFuture.get() @bci=4, line=113 (Interpreted frame)
 - org.elasticsearch.action.support.AdapterActionFuture.actionGet() @bci=1, line=45 (Interpreted frame)
 - org.elasticsearch.action.ActionRequestBuilder.get() @bci=4, line=73 (Interpreted frame)
 - org.elasticsearch.river.mongodb.MongoDBRiver.getIndexCount(org.elasticsearch.client.Client, org.elasticsearch.river.mongodb.MongoDBRiverDefinition) @bci=27, line=353 (Interpreted frame)
 - org.elasticsearch.rest.action.mongodb.RestMongoDBRiverAction.getRivers(int, int, org.elasticsearch.client.Client) @bci=339, line=173 (Interpreted frame)
 - org.elasticsearch.rest.action.mongodb.RestMongoDBRiverAction.list(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel, org.elasticsearch.client.Client) @bci=17, line=103 (Interpreted frame)
 - org.elasticsearch.rest.action.mongodb.RestMongoDBRiverAction.handleRequest(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel, org.elasticsearch.client.Client) @bci=62, line=52 (Interpreted frame)
 - org.elasticsearch.rest.BaseRestHandler.handleRequest(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel) @bci=22, line=53 (Interpreted frame)
 - org.elasticsearch.rest.RestController.executeHandler(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel) @bci=13, line=197 (Interpreted frame)
 - org.elasticsearch.rest.RestController.dispatchRequest(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel) @bci=117, line=180 (Interpreted frame)
 - org.elasticsearch.http.HttpServer.internalDispatchRequest(org.elasticsearch.http.HttpRequest, org.elasticsearch.http.HttpChannel) @bci=39, line=121 (Interpreted frame)
 - org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(org.elasticsearch.http.HttpRequest, org.elasticsearch.http.HttpChannel) @bci=6, line=83 (Interpreted frame)
 - org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(org.elasticsearch.http.HttpRequest, org.elasticsearch.http.HttpChannel) @bci=6, line=319 (Interpreted frame)
 - org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=89, line=61 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=46, line=60 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=88 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=99, line=145 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.common.netty.handler.codec.http.HttpContentDecoder.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=285, line=108 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, java.lang.Object, java.net.SocketAddress) @bci=16, line=296 (Interpreted frame)
 - org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, java.net.SocketAddress, java.lang.Object) @bci=114, line=459 (Interpreted frame)
 - org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.Channel, org.elasticsearch.common.netty.buffer.ChannelBuffer, org.elasticsearch.common.netty.buffer.ChannelBuffer, java.net.SocketAddress) @bci=176, line=536 (Interpreted frame)
 - org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=95, line=435 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=133, line=74 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=56, line=559 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(org.elasticsearch.common.netty.channel.Channel, java.lang.Object, java.net.SocketAddress) @bci=16, line=268 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(org.elasticsearch.common.netty.channel.Channel, java.lang.Object) @bci=3, line=255 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(java.nio.channels.SelectionKey) @bci=179, line=88 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(java.nio.channels.Selector) @bci=70, line=108 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run() @bci=373, line=318 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
 - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5468: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int) @bci=72, line=997 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int) @bci=24, line=1304 (Compiled frame)
 - org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get() @bci=2, line=274 (Interpreted frame)
 - org.elasticsearch.common.util.concurrent.BaseFuture.get() @bci=4, line=113 (Interpreted frame)
 - org.elasticsearch.action.support.AdapterActionFuture.actionGet() @bci=1, line=45 (Interpreted frame)
 - org.elasticsearch.action.ActionRequestBuilder.get() @bci=4, line=73 (Interpreted frame)
 - org.elasticsearch.river.mongodb.MongoDBRiver.getIndexCount(org.elasticsearch.client.Client, org.elasticsearch.river.mongodb.MongoDBRiverDefinition) @bci=27, line=353 (Interpreted frame)
 - org.elasticsearch.rest.action.mongodb.RestMongoDBRiverAction.getRivers(int, int, org.elasticsearch.client.Client) @bci=339, line=173 (Interpreted frame)
 - org.elasticsearch.rest.action.mongodb.RestMongoDBRiverAction.list(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel, org.elasticsearch.client.Client) @bci=17, line=103 (Interpreted frame)
 - org.elasticsearch.rest.action.mongodb.RestMongoDBRiverAction.handleRequest(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel, org.elasticsearch.client.Client) @bci=62, line=52 (Interpreted frame)
 - org.elasticsearch.rest.BaseRestHandler.handleRequest(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel) @bci=22, line=53 (Interpreted frame)
 - org.elasticsearch.rest.RestController.executeHandler(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel) @bci=13, line=197 (Interpreted frame)
 - org.elasticsearch.rest.RestController.dispatchRequest(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel) @bci=117, line=180 (Interpreted frame)
 - org.elasticsearch.http.HttpServer.internalDispatchRequest(org.elasticsearch.http.HttpRequest, org.elasticsearch.http.HttpChannel) @bci=39, line=121 (Interpreted frame)
 - org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(org.elasticsearch.http.HttpRequest, org.elasticsearch.http.HttpChannel) @bci=6, line=83 (Interpreted frame)
 - org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(org.elasticsearch.http.HttpRequest, org.elasticsearch.http.HttpChannel) @bci=6, line=319 (Interpreted frame)
 - org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=89, line=61 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=46, line=60 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=88 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=99, line=145 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.common.netty.handler.codec.http.HttpContentDecoder.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=285, line=108 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, java.lang.Object, java.net.SocketAddress) @bci=16, line=296 (Interpreted frame)
 - org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, java.net.SocketAddress, java.lang.Object) @bci=114, line=459 (Interpreted frame)
 - org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.Channel, org.elasticsearch.common.netty.buffer.ChannelBuffer, org.elasticsearch.common.netty.buffer.ChannelBuffer, java.net.SocketAddress) @bci=176, line=536 (Interpreted frame)
 - org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=95, line=435 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=133, line=74 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=56, line=559 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(org.elasticsearch.common.netty.channel.Channel, java.lang.Object, java.net.SocketAddress) @bci=16, line=268 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(org.elasticsearch.common.netty.channel.Channel, java.lang.Object) @bci=3, line=255 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(java.nio.channels.SelectionKey) @bci=179, line=88 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(java.nio.channels.Selector) @bci=70, line=108 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run() @bci=373, line=318 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
 - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5467: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int) @bci=72, line=997 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int) @bci=24, line=1304 (Compiled frame)
 - org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get() @bci=2, line=274 (Interpreted frame)
 - org.elasticsearch.common.util.concurrent.BaseFuture.get() @bci=4, line=113 (Interpreted frame)
 - org.elasticsearch.action.support.AdapterActionFuture.actionGet() @bci=1, line=45 (Interpreted frame)
 - org.elasticsearch.action.ActionRequestBuilder.get() @bci=4, line=73 (Interpreted frame)
 - org.elasticsearch.river.mongodb.MongoDBRiver.getIndexCount(org.elasticsearch.client.Client, org.elasticsearch.river.mongodb.MongoDBRiverDefinition) @bci=27, line=353 (Interpreted frame)
 - org.elasticsearch.rest.action.mongodb.RestMongoDBRiverAction.getRivers(int, int, org.elasticsearch.client.Client) @bci=339, line=173 (Interpreted frame)
 - org.elasticsearch.rest.action.mongodb.RestMongoDBRiverAction.list(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel, org.elasticsearch.client.Client) @bci=17, line=103 (Interpreted frame)
 - org.elasticsearch.rest.action.mongodb.RestMongoDBRiverAction.handleRequest(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel, org.elasticsearch.client.Client) @bci=62, line=52 (Interpreted frame)
 - org.elasticsearch.rest.BaseRestHandler.handleRequest(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel) @bci=22, line=53 (Interpreted frame)
 - org.elasticsearch.rest.RestController.executeHandler(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel) @bci=13, line=197 (Interpreted frame)
 - org.elasticsearch.rest.RestController.dispatchRequest(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel) @bci=117, line=180 (Interpreted frame)
 - org.elasticsearch.http.HttpServer.internalDispatchRequest(org.elasticsearch.http.HttpRequest, org.elasticsearch.http.HttpChannel) @bci=39, line=121 (Interpreted frame)
 - org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(org.elasticsearch.http.HttpRequest, org.elasticsearch.http.HttpChannel) @bci=6, line=83 (Interpreted frame)
 - org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(org.elasticsearch.http.HttpRequest, org.elasticsearch.http.HttpChannel) @bci=6, line=319 (Interpreted frame)
 - org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=89, line=61 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=46, line=60 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=88 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=99, line=145 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.common.netty.handler.codec.http.HttpContentDecoder.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=285, line=108 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, java.lang.Object, java.net.SocketAddress) @bci=16, line=296 (Interpreted frame)
 - org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, java.net.SocketAddress, java.lang.Object) @bci=114, line=459 (Interpreted frame)
 - org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.Channel, org.elasticsearch.common.netty.buffer.ChannelBuffer, org.elasticsearch.common.netty.buffer.ChannelBuffer, java.net.SocketAddress) @bci=176, line=536 (Interpreted frame)
 - org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=95, line=435 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=70 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=133, line=74 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=56, line=559 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(org.elasticsearch.common.netty.channel.Channel, java.lang.Object, java.net.SocketAddress) @bci=16, line=268 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(org.elasticsearch.common.netty.channel.Channel, java.lang.Object) @bci=3, line=255 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(java.nio.channels.SelectionKey) @bci=179, line=88 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(java.nio.channels.Selector) @bci=70, line=108 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run() @bci=373, line=318 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
 - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5466: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int) @bci=72, line=997 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int) @bci=24, line=1304 (Compiled frame)
 - org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get() @bci=2, line=274 (Interpreted frame)
 - org.elasticsearch.common.util.concurrent.BaseFuture.get() @bci=4, line=113 (Interpreted frame)
 - org.elasticsearch.action.support.AdapterActionFuture.actionGet() @bci=1, line=45 (Interpreted frame)
 - org.elasticsearch.action.ActionRequestBuilder.get() @bci=4, line=73 (Interpreted frame)
 - org.elasticsearch.river.mongodb.MongoDBRiver.getIndexCount(org.elasticsearch.client.Client, org.elasticsearch.river.mongodb.MongoDBRiverDefinition) @bci=27, line=353 (Interpreted frame)
 - org.elasticsearch.rest.action.mongodb.RestMongoDBRiverAction.getRivers(int, int, org.elasticsearch.client.Client) @bci=339, line=173 (Interpreted frame)
 - org.elasticsearch.rest.action.mongodb.RestMongoDBRiverAction.list(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel, org.elasticsearch.client.Client) @bci=17, line=103 (Interpreted frame)
 - org.elasticsearch.rest.action.mongodb.RestMongoDBRiverAction.handleRequest(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel, org.elasticsearch.client.Client) @bci=62, line=52 (Interpreted frame)
 - org.elasticsearch.rest.BaseRestHandler.handleRequest(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel) @bci=22, line=53 (Interpreted frame)
 - org.elasticsearch.rest.RestController.executeHandler(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel) @bci=13, line=197 (Interpreted frame)
 - org.elasticsearch.rest.RestController.dispatchRequest(org.elasticsearch.rest.RestRequest, org.elasticsearch.rest.RestChannel) @bci=117, line=180 (Interpreted frame)
 - org.elasticsearch.http.HttpServer.internalDispatchRequest(org.elasticsearch.http.HttpRequest, org.elasticsearch.http.HttpChannel) @bci=39, line=121 (Interpreted frame)
 - org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(org.elasticsearch.http.HttpRequest, org.elasticsearch.http.HttpChannel) @bci=6, line=83 (Interpreted frame)
 - org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(org.elasticsearch.http.HttpRequest, org.elasticsearch.http.HttpChannel) @bci=6, line=319 (Interpreted frame)
 - org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=89, line=61 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=70 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=46, line=60 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=88 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=99, line=145 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=70 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.common.netty.handler.codec.http.HttpContentDecoder.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=285, line=108 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=70 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, java.lang.Object, java.net.SocketAddress) @bci=16, line=296 (Interpreted frame)
 - org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, java.net.SocketAddress, java.lang.Object) @bci=114, line=459 (Interpreted frame)
 - org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.Channel, org.elasticsearch.common.netty.buffer.ChannelBuffer, org.elasticsearch.common.netty.buffer.ChannelBuffer, java.net.SocketAddress) @bci=176, line=536 (Interpreted frame)
 - org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.MessageEvent) @bci=95, line=435 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=13, line=70 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=22, line=791 (Compiled frame)
 - org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(org.elasticsearch.common.netty.channel.ChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=133, line=74 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext, org.elasticsearch.common.netty.channel.ChannelEvent) @bci=9, line=564 (Compiled frame)
 - org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(org.elasticsearch.common.netty.channel.ChannelEvent) @bci=56, line=559 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(org.elasticsearch.common.netty.channel.Channel, java.lang.Object, java.net.SocketAddress) @bci=16, line=268 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(org.elasticsearch.common.netty.channel.Channel, java.lang.Object) @bci=3, line=255 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(java.nio.channels.SelectionKey) @bci=179, line=88 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(java.nio.channels.Selector) @bci=70, line=108 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run() @bci=373, line=318 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
 - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5465: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=177, line=734 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1273 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, line=1066 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5464: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=442 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5462: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
 - org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.waitForNextTick() @bci=81, line=445 (Compiled frame)
 - org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.run() @bci=43, line=364 (Interpreted frame)
 - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5459: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2039 (Interpreted frame)
 - java.util.concurrent.PriorityBlockingQueue.take() @bci=22, line=549 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5458: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Interpreted frame)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Interpreted frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Interpreted frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Interpreted frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Interpreted frame)
 - sun.nio.ch.SelectorImpl.select() @bci=2, line=101 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioServerBoss.select(java.nio.channels.Selector) @bci=1, line=163 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioServerBoss.run() @bci=1, line=42 (Interpreted frame)
 - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5457: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=68 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(java.nio.channels.Selector) @bci=1, line=415 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
 - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5456: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=68 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(java.nio.channels.Selector) @bci=1, line=415 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
 - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5455: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=68 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(java.nio.channels.Selector) @bci=1, line=415 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
 - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5454: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=68 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(java.nio.channels.Selector) @bci=1, line=415 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
 - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5453: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=68 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(java.nio.channels.Selector) @bci=1, line=415 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run() @bci=1, line=42 (Interpreted frame)
 - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5452: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=68 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(java.nio.channels.Selector) @bci=1, line=415 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
 - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5451: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=68 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(java.nio.channels.Selector) @bci=1, line=415 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
 - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5450: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=68 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(java.nio.channels.Selector) @bci=1, line=415 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
 - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5449: (state = IN_NATIVE)
 - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise)
 - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=79 (Compiled frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 (Compiled frame)
 - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=68 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(java.nio.channels.Selector) @bci=1, line=415 (Compiled frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run() @bci=56, line=212 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=1, line=89 (Interpreted frame)
 - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=178 (Interpreted frame)
 - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=108 (Interpreted frame)
 - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1142 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5448: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=97, line=2163 (Interpreted frame)
 - org.elasticsearch.indices.ttl.IndicesTTLService$Notifier.await() @bci=21, line=325 (Interpreted frame)
 - org.elasticsearch.indices.ttl.IndicesTTLService$PurgerThread.run() @bci=70, line=147 (Interpreted frame)

Thread 5447: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=177, line=734 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Interpreted frame)
 - java.util.concurrent.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1273 (Interpreted frame)
 - org.elasticsearch.cluster.action.index.MappingUpdatedAction$MasterMappingUpdater.run() @bci=21, line=382 (Interpreted frame)

Thread 5446: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=124, line=1093 (Compiled frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=809 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

Thread 5445: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
 - org.elasticsearch.threadpool.ThreadPool$EstimatedTimeThread.run() @bci=18, line=567 (Interpreted frame)

Thread 5440: (state = BLOCKED)

Thread 5439: (state = BLOCKED)

Thread 5438: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=142 (Compiled frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=158 (Compiled frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=209 (Interpreted frame)

Thread 5437: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait() @bci=2, line=502 (Interpreted frame)
 - java.lang.ref.Reference$ReferenceHandler.run() @bci=36, line=157 (Interpreted frame)

ankon commented 9 years ago

The river is indeed stuck in start(), but the line numbers don't match up with what I would expect for master.

When the river is starting it prints its version into the elasticsearch log file, could you try to find those lines and add them here? They look like this:

[2014-12-16 17:41:00,636][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB River Plugin - version[2.0.5-rc-collaborne-4] - hash[0fb952c] - time[2014-12-16T10:38:36Z]

Also, you used jstack -F, I guess that was because a regular jstack didn't work and suggested that?

tmatei commented 9 years ago

@ankon You're totally right I was really tired last night and installed the wrong plugin. I had an old version of it with 2.0.3 when I was testing it earlier and it autocompleted and I just used it.

[2014-12-16 21:35:37,286][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB River Plugin - version[2.0.3-SNAPSHOT] - hash[0e34d15] - time[2014-11-09T05:48:44Z]

The good news is that it works great! awesome work! Thanks!

benmccann commented 9 years ago

I'd suggest testing on 2.0.5, which was recently released with numerous bug fixes. Also, does the info in https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/442#issuecomment-67532291 help you?

rajeeshv commented 9 years ago

I had tried ES 1.4, River downloaded from master packaged using maven, MongoDB 2.6.5. Not worked. Should the version of MongoDB be 2.6.6 ?

benmccann commented 9 years ago

Yes, Mongo 2.6.6 works fine. Other folks have said when theirs wasn't working that uninstalling and re-installing various components helped. You might want to look through the other bugs including ones that were recently closed

richardwilly98 / elasticsearch-river-mongodb

elastic search river mongodb not indexing the documents #428