search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
67 stars 3.5k forks source link

Logstash 1:1.5.0-1 dies after some time #3276

Closed parabolic closed 7 years ago

parabolic commented 9 years ago

Hi, today I've upgraded to logstash version 1:1.5.0-1 on ubuntu 14.04, x86_64, and it seems to hangs after some time without any notification in the logs whatsoever. When I try to stop the process with issuing 

service logstash stop

or 

/etc/init.d/logstash stop

I get the following:

killing logstash (pid 26022) with SIGTERM
Waiting logstash (pid 26022) to die...
Waiting logstash (pid 26022) to die...
Waiting logstash (pid 26022) to die...
Waiting logstash (pid 26022) to die...
Waiting logstash (pid 26022) to die...
logstash stop failed; still running.
logstash started.

and then I have to do a 

kill -9

to kill the process and it goes on and on. I started logstash manually with the debug option and after some time it died again but I got the following line at the end

@metadata_accessors=#<LogStash::Util::Accessors:0x2513a1f1 @store={"retry_count"=>0}, @lut={}>, @cancelled=false>]]}, :batch_timeout=>1, :force=>true, :final=>nil, :level=>:debug, :file=>"stud/buffer.rb", :line=>"207", :method=>"buffer_flush"}
Sending bulk of actions to client[0]: localhost {:level=>:debug, :file=>"logstash/outputs/elasticsearch.rb", :line=>"461", :method=>"flush"}
Shifting current elasticsearch client {:level=>:debug, :file=>"logstash/outputs/elasticsearch.rb", :line=>"468", :method=>"flush"}
Switched current elasticsearch client to #0 at localhost {:level=>:debug, :file=>"logstash/outputs/elasticsearch.rb", :line=>"518", :method=>"shift_client"}

The debug output stops at this point and the process hangs.

I have a lot of logshtash "senders" which just use lumberjack to the main node to send encrypted logs and they are upgraded to 1.5 and they work fine. It seems that the "master" logstash with elasticsearch output has a problem somewhere which is causing it to hang. here's the output config

elasticsearch {
      host => "localhost"
      protocol => "http"
    }

At the moment I've reverted back to logstash 1.4.2 Thanks and regards.

ph commented 9 years ago

@parabolic Would you mind posting the configuration from your logstash master?

parabolic commented 9 years ago

No problem but I will have to ommit the filter and some names just give me some time.

parabolic commented 9 years ago

Ok here it is, mind me but the filter section is quite big and we only use the lumberjack input the other input is just there for compatibility but will be removed soon.

input   {
  udp {
    type => syslog
    tags => syslog
    port => someport_number
  }

  ## SSL encrypted input
  lumberjack {
     port             => some_other_port_number
     ssl_certificate  => "/path/to/ssl/certs/cert.pem"
     ssl_key          => "/path/to/ssl/certs/cert.key"
     tags             => syslog
     type             => syslog
  }
}

filter  {

}

output {
  if [tags] == “some_tags” {
    #stdout { codec => rubydebug }
    file {
      message_format => "%{[message][0]}"
      path => “/some/path/some_name-%{+YYYY-MM-dd}.txt"
    }
    elasticsearch {
      host => "localhost"
      protocol => "http"
    }
  }
  if [type] == “syslog” {
   #stdout { codec => rubydebug }
    elasticsearch {
      host => "localhost"
      protocol => "http"
    }
  }
}
ph commented 9 years ago

Do you have any errors on the elasticsearch side log?

parabolic commented 9 years ago

I have a failed to parse query, from kibana4 and I have a failed recovery shard, due to restart of ES a few days ago. Here's the error:

reason [shard failure [failed recovery][IndexShardGatewayRecoveryException[[kibana-int][2] failed recovery]; nested: EngineCreationFailureException[[kibana-int][2] failed to upgrade 3x segments]; nested: EOFException[read past EOF: NIOFSIndexInput(path="/some/pats/elasticsearch/nodes/0/indices/kibana-int/2/index/segments_8y")]; ]]
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [kibana-int][2] failed recovery
        at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:162)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [kibana-int][2] failed to upgrade 3x segments
        at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:121)
ph commented 9 years ago

Can you list your filter your are using too?

parabolic commented 9 years ago

Hi yes, I've ommited the tags and comments and some matches,

filter  {

  if [type] == "syslog" {
    grok {

      match => { "message" => ""}
      match => { "message" => ""}
      match => { "message" => ""}
      match => { "message" => ""}
      match => { "message" => ""}
      match => { "message" => ""}
      match => { "message" => ""}
     match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:syslog_hostname} %{SYSLOGPROG:syslog_program}: %{DATA:severity}: %{GREEDYDATA:message_data}"}
      ## Syslog better match
      match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:syslog_hostname} %{SYSLOGPROG:syslog_program} %{GREEDYDATA:message_data}"}
      match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:syslog_hostname} %{SYSLOGPROG:syslog_program}: %{GREEDYDATA:message_data}"}
      match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:syslog_hostname} %{SYSLOGPROG:syslog_program}%{GREEDYDATA:message_data}"}
      ## Everything else matching
      match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{HOSTNAME:syslog_hostname} %{GREEDYDATA:message_data}"}
    }

    if [syslog_program] == "web_server"{
      date {
        match => ["web_server_timestamp", "ISO8601"]
      }
    } else {
      date {
        match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
      }
    }

    if [syslog_program] == "some_tag" {
      multiline {
        pattern => "(\]\[DEBUG\]\[|\]\[EMERGENCY\]\[|\]\[ALERT\]\[|\]\[CRITICAL\]\[|\]\[ERROR\]\[|\]\[WARNING\]\[|\]\[NOTICE\]\[|\]\[INFORMATIONAL\]\[|\]\[TRACE\]\[)"
        what    => "previous"
        negate  => true
      }
    }
    if [syslog_program] == “web_server”{
      useragent {
        source      => "some_tag"
      }
           if [some_tag] == “some_match” or [some_tag] == "some_match" or [some_tag] == "some_match" or [some_tag] == "some_match" or [some_tag] == "some_match" or [some_tag] == "some_match" or [some_tag] == "some_match" {
        mutate {
          split => [“some_tag”, ","]
        }
        mutate {
          add_field => {
            “some_tag”              => "%{[some_tag][0]}"
            "some_tag" => "%{[some_tag][-1]}"
            "some_tag"   => "%{some_tag}"
          }
        }
      }
      mutate {
        strip => ["some_tag", “some_string” ]
      }
      mutate {
        add_field => {
          “region” => “%{some_tag}”
        }
        add_field => {
          "os_full_name" => “%{some_tag}”
        }
        convert => [ "some_tag", "float" ]
        convert => [ "some_tag", "integer" ]
        convert => [ "some_tag", "integer" ]
        convert => [ "some_tag", "integer" ]
        convert => [ "some_tag", "integer" ]
        convert => [ "some_tag", "integer" ]
        convert => [ "some_tag", "integer" ]
        gsub => [
        "some_tag","\"","",
        "some_tag","\"","",
        "some_tag","\"","",
        "some_tag","\"","",
        "some_tag","\"",""
        ]
      }
      if [some_tag] =~ /^([1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(\.([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])){3}(?:\-([1-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]))?$/ {
        geoip {
          source => "some_tag"
        }
      } else {
        mutate {
          replace => [ "some_tag", "0.0.0.0"]
        }
      }
      if [some_tag] =~ /\?/ {
        mutate {
          split => ["some_tag", "?"]
        }
        mutate {
          add_field => {
            request_path => "%{[some_tag][0]}"
          }
        }
        ## After we've taken the request path we merge the whole uri again from the array
        mutate {
          join => ["some_tag","?"]
        }
      } else {
        mutate {
          add_field => {
            request_path => "%{some_tag}"
          }
        }
      }
      if [some_tag] =~ /some_secret_field/ {
        mutate {
          remove_field => [ "some_tag" ]
          remove_field => [ "some_tag" ]
        }
      }
      mutate {
        replace => [ "some_tag", "%{some_tag}"]
      }
    } else {
      mutate {
        add_field => {
          "some_tag" => "%{some_tag}"
        }
      }
    }
    if [some_tag] =~ /some_string/ {
      mutate {
        add_field => {
          "some_tag" =>  ""
          "some_tag" => ""
          "some_tag" => ""
        }
      }
      mutate {
        split => ["some_tag", "_"]
      }
      mutate {
        update => [ "some_tag", "%{[some_tag][1]}" ]
        update => [ "some_tag", "%{[some_tag][2]}" ]
        update => [ "some_tag", "%{[some_tag][0]}" ]
        update => [ "some_tag", "%{[some_tag][0]}" ]
      }
      mutate {
        split => ["some_tag", "."]
      }
      mutate {
        update => [ "some_tag", "%{[some_tag][0]}" ]
      }
    }
    mutate {
      split => ["some_tag", "-"]
    }
    mutate {
      update => [ "some_tag", "%{[some_tag][0]}" ]
      ## Removing everything into the discard field
      remove_field => [ "discard", "some_tag", "some_tag", "some_tag", "some_tag", "some_tag" ]
    }
    ## Parse the syslog pri code
    syslog_pri {
    }
  }
}
ph commented 9 years ago

Maybe there is a dead lock on the es output, can you do jstack PID and paste the content in a gist?

parabolic commented 9 years ago

The thing is that I've disabled it completely and reverted back to 1.4.2, I can gather logs and nc them to an logstash 1.5 instance running with the same config file and will do the jstack PID and hope that it will hang again. So just to clarify, it stops without any error in the log file, we are sending massive amounts of logs, the only thing that I got was the debug message that I've pasted above. Before that debug message/error there was a normal log displayed. After kill -9 PID and service logstash start it was fine again for some time.

parabolic commented 9 years ago

Hi, I've added 2 gists. 1 is when logstash is just started and the other is when I it's under load. I've simulated load with a while loop and logger that sends a simple text directly to logstash and it didn't hang. I've used the same config as the production one just changed the ports and it's running on the same server. The other thing is that I've used the UDP listener to send the logs to and not the lumberjack one. Here are the gists: https://gist.github.com/parabolic/7fbb7c856f72406311f1 https://gist.github.com/parabolic/7a07bc115858d4f4516a

and here is the event in kibana

2015-05-19 21-14-35

Logstash is still running fine but since I've started it with --debug I get this every 5 seconds

Flushing {:plugin=><LogStash::Filters::Multiline pattern=>"(\\]\\[DEBUG\\]\\[|\\]\\[EMERGENCY\\]\\[|\\]\\[ALERT\\]\\[|\\]\\[CRITICAL\\]\\[|\\]\\[ERROR\\]\\[|\\]\\[WARNING\\]\\[|\\]\\[NOTICE\\]\\[|\\]\\[INFORMATIONAL\\]\\[|\\]\\[TRACE\\]\\[)", what=>"previous", source=>"message", stream_identity=>"%{host}.%{path}.%{type}">, :level=>:debug, :file=>"(eval)", :line=>"127", :method=>"initialize"}
ph commented 9 years ago

@parabolic Thank you I'll do more testing on my side, did you try to run 1.5 side by side with nc?

parabolic commented 9 years ago

@ph I am unsure what are you asking me here, the above response is logstash 1.5 with nc sending simple string and delivering it to elasticsearch. It's just started manually from the command line. I think something in the filter is killing it. Thanks for looking into this.

Centopus commented 9 years ago

Hello. I seem to be having the same issue. Running the stable version of 1.5.0 (the one that is served from the main site - CHANGELOG file says: 1.5.0 (May 14, 2015)). (jdk1.8.0_45, elasticsearch-1.5.1) I've started a topic over on discuss.elastic.co. https://discuss.elastic.co/t/logstash-stops-processing-files-after-a-while/944

To summarize: Randomly stops processing input, when in debug mode '_discover_file_glob:' stop showing up for the files that change, after a while it checks stale files only. Cant kill it (need kill -9 to kill). Can make it stop, by sending node shutdown request from elasticsearch cluster side.

The 'random' stop-reading-input moment seems to be more likely when larger spikes of data come in, but I'm unsure about that.

I've copied my config here, not going to copy anything else, unless asked for.

input {

  file {
        type => "WLlog"
        path => ["/TEST/logs/AdminServer/AdminServer.log"]
        tags => [TEST,Admin,log]
       }
  file {
        type => "WLlog"
        path => ["/PROD/logs/AdminServer/AdminServer.log"]
        tags => [PROD,Admin,log]
       }
  file {
        type => "WLlog"
        path => ["/TEST/logs/AdminServer/TEST.log"]
        tags => [TEST,Domain,log]
        }
  file {
        type => "WLlog"
        path => ["/PROD/logs/AdminServer/PROD.log"]
        tags => [PROD,Domain,log]
        }
  file {
        type => "WLlog"
        path => [
                        "/TEST/logs/osb_server1/osb_server1.log",
                        "/TEST/logs/osb_server2/osb_server2.log",
                        "/TEST/logs/osb_server3/osb_server3.log",
                        "/TEST/logs/osb_server4/osb_server4.log",
                        "/TEST/logs/osb_server5/osb_server5.log",
                        "/TEST/logs/osb_server6/osb_server6.log"
                ]
        tags => [TEST,OSB,log]
        }
 file {
        type => "WLlog"
        path => [
                        "/PROD/logs/osb_server1/osb_server1.log",
                        "/PROD/logs/osb_server2/osb_server2.log",
                        "/PROD/logs/osb_server3/osb_server3.log",
                        "/PROD/logs/osb_server4/osb_server4.log",
                        "/PROD/logs/osb_server5/osb_server5.log",
                        "/PROD/logs/osb_server6/osb_server6.log"
                ]
        tags => [PROD,OSB,log]
        }
  file {
        type => "WLlog"
        path => [
                        "/TEST/logs/soa_server1/soa_server1.log",
                        "/TEST/logs/soa_server2/soa_server2.log",
                        "/TEST/logs/soa_server3/soa_server3.log",
                        "/TEST/logs/soa_server4/soa_server4.log",
                        "/TEST/logs/soa_server5/soa_server5.log",
                        "/TEST/logs/soa_server6/soa_server6.log"
                ]
        tags => [TEST,SOA,log]
        }
  file {
        type => "WLlog"
        path => [
                        "/PROD/logs/soa_server1/soa_server1.log",
                        "/PROD/logs/soa_server2/soa_server2.log",
                        "/PROD/logs/soa_server3/soa_server3.log",
                        "/PROD/logs/soa_server4/soa_server4.log",
                        "/PROD/logs/soa_server5/soa_server5.log",
                        "/PROD/logs/soa_server6/soa_server6.log"
                ]
        tags => [PROD,SOA,log]
        }
  file {
        type => "TRlog"
        path => ["/TEST/otdcfg/inst-1/TEST/logs/*.log"]
        tags => [TEST,traf,log,vtd1]
       }
  file {
        type => "TRlog"
        path => ["/TEST/otdcfg/inst-2/TEST/logs/*.log"]
        tags => [TEST,traf,log,vtd2]
       }
  file {
        type => "TRlog"
        path => ["/PROD/otdcfg/inst-1/PROD/logs/*.log"]
        tags => [PROD,traf,log,vtd1]
       }
  file {
        type => "TRlog"
        path => ["/PROD/otdcfg/inst-2/PROD/logs/*.log"]
        tags => [PROD,traf,log,vtd2]
       }
  file {
        type => "NMlog"
        path => [
                        "/TEST/nodemgrs/TEST1/logs/nodemanager.log",
                        "/TEST/nodemgrs/TEST2/logs/nodemanager.log",
                        "/TEST/nodemgrs/TEST3/logs/nodemanager.log",
                        "/TEST/nodemgrs/TEST4/logs/nodemanager.log",
                        "/TEST/nodemgrs/TEST5/logs/nodemanager.log",
                        "/TEST/nodemgrs/TEST6/logs/nodemanager.log"
                ]
        tags => [TEST,NM,log]
        }
  file {
        type => "NMlog"
        path => [
                        "/PROD/nodemgrs/PROD1/logs/nodemanager.log",
                        "/PROD/nodemgrs/PROD2/logs/nodemanager.log",
                        "/PROD/nodemgrs/PROD3/logs/nodemanager.log",
                        "/PROD/nodemgrs/PROD4/logs/nodemanager.log",
                        "/PROD/nodemgrs/PROD5/logs/nodemanager.log",
                        "/PROD/nodemgrs/PROD6/logs/nodemanager.log"
                ]
        tags => [PROD,NM,log]
        }
}

filter {
  if [type] == "WLlog"
  {
        multiline {
                   pattern => "^####"
                   negate => true
                   what => "previous"
                   }
        grok {
             match => [ "message", "####<%{DATA:wls_timestamp}> <%{WORD:severity}> <%{DATA:wls_topic}> <%{HOST:hostname}> <(%{WORD:server})?> %{GREEDYDATA:logmessage}"]
             }
  }
  if [type] == "TRlog"
                {
                grok    {
                        match => [ "message", "%{IP:sourceIP} - - \[%{DATA:timestamp}\] \"%{DATA:action}\" %{NUMBER:code} %{NUMBER:size} %{DATA:messageID} %{HOST:hostname}:%{NUMBER:port}"]
                        }
                }
}

output { elasticsearch { host => "elastic_machine"
                         cluster => FreeWorldAlliance
                       }
               }
ph commented 9 years ago

@Centopus Can you give us a thread dump with jstack? I am investigating #3302 and I think its related to you issue I am trying hard to nail the problem. If you are able to reproduce the bug can you add this ruby filter to before your other filter? This will create a log of noise in your log but will give us the last event before the hang. Thank you

filter {
  ruby {
    code => "@logger.warn(event.to_hash)"
  }
}
Centopus commented 9 years ago

I'll do it first thing in the morning on Monday (GMT+2). Unfortunately I do not have access to that environment now.

ph commented 9 years ago

@Centopus can you get a heap dump too? Thank you very much.

celesteking commented 9 years ago

All people having troubles, could you make a netstat -pnt|grep java on logstash host and see if there're any numbers > 0 in recv/transmit queue.

It appears I'm hitting the same issue with high traffic parallel lumberjack connections.

neilprosser commented 9 years ago 
# netstat -pnt|grep java
tcp        0      0 ::ffff:10.216.142.16:50440  ::ffff:10.216.141.35:6379   ESTABLISHED 7690/java
tcp        1      0 ::ffff:10.216.142.16:39316  ::ffff:10.216.142.92:80     CLOSE_WAIT  7690/java
tcp        0      0 ::ffff:10.216.142.16:50460  ::ffff:10.216.141.35:6379   ESTABLISHED 7690/java
tcp        1      0 ::ffff:10.216.142.16:49210  ::ffff:10.216.142.174:80    CLOSE_WAIT  7690/java
tcp        0      0 ::ffff:10.216.142.16:50444  ::ffff:10.216.141.35:6379   ESTABLISHED 7690/java
tcp        1      0 ::ffff:10.216.142.16:39318  ::ffff:10.216.142.92:80     CLOSE_WAIT  7690/java
tcp        0      0 ::ffff:10.216.142.16:50441  ::ffff:10.216.141.35:6379   ESTABLISHED 7690/java
tcp        1      0 ::ffff:10.216.142.16:39319  ::ffff:10.216.142.92:80     CLOSE_WAIT  7690/java

I'll put this into the issue I started too.

Centopus commented 9 years ago

Thread dump after trying to shut it down. Restarting now, and waiting for it to crash to provide the other requested data.

And yes.. i just noticed, you wanted a heap dump, not a thread dump... working on it now, I'll leave the thead dump anyway, might be of some use.

//EDIT: // Its running fine for the last 3h... I'll get back to you when it hangs...

SIGINT received. Shutting down the pipeline. {:level=>:warn}
Sending shutdown signal to input thread {:thread=>#<Thread:0x4e541911 dead>, :level=>:info}
Sending shutdown signal to input thread {:thread=>#<Thread:0x3ba2e91e dead>, :level=>:info}
Sending shutdown signal to input thread {:thread=>#<Thread:0xbc68dce dead>, :level=>:info}
Sending shutdown signal to input thread {:thread=>#<Thread:0x28ccfcc dead>, :level=>:info}
Sending shutdown signal to input thread {:thread=>#<Thread:0x4fa763c dead>, :level=>:info}
Sending shutdown signal to input thread {:thread=>#<Thread:0x8b2a4b5 dead>, :level=>:info}
Sending shutdown signal to input thread {:thread=>#<Thread:0x435ff586 dead>, :level=>:info}
Sending shutdown signal to input thread {:thread=>#<Thread:0x436b2230 dead>, :level=>:info}
Sending shutdown signal to input thread {:thread=>#<Thread:0x4177b292 dead>, :level=>:info}
Sending shutdown signal to input thread {:thread=>#<Thread:0x3f920635 dead>, :level=>:info}
Sending shutdown signal to input thread {:thread=>#<Thread:0x458ce848 dead>, :level=>:info}
Sending shutdown signal to input thread {:thread=>#<Thread:0x3e50187f run>, :level=>:info}
Sending shutdown signal to input thread {:thread=>#<Thread:0x2e778ea9 dead>, :level=>:info}
Sending shutdown signal to input thread {:thread=>#<Thread:0x4d299c44 dead>, :level=>:info}
2015-05-25 07:10:09
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode):

"Ruby-0-JIT-5" #78 daemon prio=1 os_prio=0 tid=0x00000000025ee800 nid=0x4ab1 waiting on condition [0x00007f1361fee000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000eb849608> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][management][T#3]" #76 daemon prio=5 os_prio=0 tid=0x0000000002f7a800 nid=0x4aa4 waiting on condition [0x00007f13601e1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ec53b258> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:734)
        at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
        at java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1273)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][management][T#2]" #72 daemon prio=5 os_prio=0 tid=0x00000000028a3800 nid=0x5131 waiting on condition [0x00007f13621ef000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ec53b258> (a org.elasticsearch.common.util.concurrent.EsExecutors$ExecutorScalingQueue)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:734)
        at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
        at java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1273)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

">output" #70 daemon prio=5 os_prio=0 tid=0x00007f137915b000 nid=0x5124 in Object.wait() [0x00007f13603e2000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:460)
        at org.jruby.RubyThread$SleepTask.run(RubyThread.java:1049)
        - locked <0x00000000ed56e3d0> (a org.jruby.ext.thread.SizedQueue)
        at org.jruby.RubyThread.executeBlockingTask(RubyThread.java:1065)
        at org.jruby.RubyThread.wait_timeout(RubyThread.java:1413)
        at org.jruby.ext.thread.Queue.pop(Queue.java:152)
        - eliminated <0x00000000ed56e3d0> (a org.jruby.ext.thread.SizedQueue)
        at org.jruby.ext.thread.Queue.pop(Queue.java:127)
        - eliminated <0x00000000ed56e3d0> (a org.jruby.ext.thread.SizedQueue)
        at org.jruby.ext.thread.SizedQueue.pop(SizedQueue.java:111)
        - locked <0x00000000ed56e3d0> (a org.jruby.ext.thread.SizedQueue)
        at org.jruby.ext.thread.SizedQueue$INVOKER$i$pop.call(SizedQueue$INVOKER$i$pop.gen)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
        at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)
        at org.jruby.ast.LocalAsgnNode.interpret(LocalAsgnNode.java:123)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.ast.WhileNode.interpret(WhileNode.java:131)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:139)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:187)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
        at org.jruby.ast.VCallNode.interpret(VCallNode.java:88)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
        at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
        at org.jruby.runtime.Block.call(Block.java:101)
        at org.jruby.RubyProc.call(RubyProc.java:290)
        at org.jruby.RubyProc.call(RubyProc.java:228)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)
        at java.lang.Thread.run(Thread.java:745)

"Ruby-0-Thread-22: /logstash/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.2.4-java/lib/logstash/outputs/elasticsearch.rb:377" #69 daemon prio=5 os_prio=0 tid=0x00007f137915d800 nid=0x5123 in Object.wait() [0x00007f13605e2000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:460)
        at org.jruby.RubyThread$SleepTask.run(RubyThread.java:1049)
        - locked <0x00000000edc64b20> (a org.jruby.ext.thread.ConditionVariable)
        at org.jruby.RubyThread.executeBlockingTask(RubyThread.java:1065)
        at org.jruby.RubyThread.wait_timeout(RubyThread.java:1413)
        at org.jruby.ext.thread.ConditionVariable.wait_ruby(ConditionVariable.java:96)
        - locked <0x00000000edc64b20> (a org.jruby.ext.thread.ConditionVariable)
        at org.jruby.ext.thread.ConditionVariable$INVOKER$i$0$1$wait_ruby.call(ConditionVariable$INVOKER$i$0$1$wait_ruby.gen)
        at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:665)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:206)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
        at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
        at org.jruby.runtime.Block.yield(Block.java:142)
        at org.jruby.ext.thread.Mutex.synchronize(Mutex.java:149)
        at org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
        at org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.ast.WhileNode.interpret(WhileNode.java:131)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
        at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
        at org.jruby.runtime.Block.call(Block.java:101)
        at org.jruby.RubyProc.call(RubyProc.java:290)
        at org.jruby.RubyProc.call(RubyProc.java:228)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)
        at java.lang.Thread.run(Thread.java:745)

"Ruby-0-Thread-21: /logstash/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-0.2.4-java/lib/logstash/outputs/elasticsearch.rb:370" #68 daemon prio=5 os_prio=0 tid=0x00007f1378c8f000 nid=0x5122 in Object.wait() [0x00007f13607e3000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.jruby.RubyThread.sleep(RubyThread.java:1001)
        - locked <0x00000000ec9bf620> (a org.jruby.RubyThread)
        - locked <0x00000000ec9bf620> (a org.jruby.RubyThread)
        at org.jruby.RubyKernel.sleep(RubyKernel.java:834)
        at org.jruby.RubyKernel$INVOKER$s$0$1$sleep.call(RubyKernel$INVOKER$s$0$1$sleep.gen)
        at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:665)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:206)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
        at org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
        at org.jruby.runtime.Interpreted19Block.yieldSpecific(Interpreted19Block.java:130)
        at org.jruby.runtime.Block.yieldSpecific(Block.java:111)
        at org.jruby.RubyKernel.loop(RubyKernel.java:1507)
        at org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(RubyKernel$INVOKER$s$0$0$loop.gen)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:316)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:145)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
        at org.jruby.ast.FCallNoArgBlockNode.interpret(FCallNoArgBlockNode.java:32)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
        at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
        at org.jruby.runtime.Block.call(Block.java:101)
        at org.jruby.RubyProc.call(RubyProc.java:290)
        at org.jruby.RubyProc.call(RubyProc.java:228)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)
        at java.lang.Thread.run(Thread.java:745)

"Ruby-0-Thread-20: /logstash/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/buffer.rb:92" #67 daemon prio=5 os_prio=0 tid=0x00007f1378981000 nid=0x5121 in Object.wait() [0x00007f13609e4000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.jruby.RubyThread.sleep(RubyThread.java:1001)
        - locked <0x00000000ed4f2f80> (a org.jruby.RubyThread)
        - locked <0x00000000ed4f2f80> (a org.jruby.RubyThread)
        at org.jruby.RubyKernel.sleep(RubyKernel.java:834)
        at org.jruby.RubyKernel$INVOKER$s$0$1$sleep.call(RubyKernel$INVOKER$s$0$1$sleep.gen)
        at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:665)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:206)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
        at org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
        at org.jruby.runtime.Interpreted19Block.yieldSpecific(Interpreted19Block.java:130)
        at org.jruby.runtime.Block.yieldSpecific(Block.java:111)
        at org.jruby.RubyKernel.loop(RubyKernel.java:1507)
        at org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(RubyKernel$INVOKER$s$0$0$loop.gen)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:316)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:145)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
        at org.jruby.ast.FCallNoArgBlockNode.interpret(FCallNoArgBlockNode.java:32)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
        at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
        at org.jruby.runtime.Block.call(Block.java:101)
        at org.jruby.RubyProc.call(RubyProc.java:290)
        at org.jruby.RubyProc.call(RubyProc.java:228)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][listener][T#1]" #66 daemon prio=5 os_prio=0 tid=0x0000000002cfa800 nid=0x5120 waiting on condition [0x00007f1360be6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ed1c07b8> (a java.util.concurrent.LinkedTransferQueue)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737)
        at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
        at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1265)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][transport_client_timer][T#1]{Hashed wheel timer #1}" #44 daemon prio=5 os_prio=0 tid=0x0000000002ffe800 nid=0x511b waiting on condition [0x00007f13617ea000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:445)
        at org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:364)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][generic][T#1]" #56 daemon prio=5 os_prio=0 tid=0x0000000002fec800 nid=0x5114 waiting on condition [0x00007f13625f1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ed1c1b38> (a java.util.concurrent.SynchronousQueue$TransferStack)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][clusterService#updateTask][T#1]" #55 daemon prio=5 os_prio=0 tid=0x00007f13641a7800 nid=0x5113 waiting on condition [0x00007f13627f2000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000eda4a240> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][[http_server_boss.default]][T#1]{New I/O server boss #18}" #54 daemon prio=5 os_prio=0 tid=0x0000000002549000 nid=0x5112 runnable [0x00007f13629f3000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ee6597e0> (a sun.nio.ch.Util$2)
        - locked <0x00000000ee6597f8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ee6640e8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.elasticsearch.common.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
        at org.elasticsearch.common.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][[http_server_worker.default]][T#8]{New I/O worker #17}" #53 daemon prio=5 os_prio=0 tid=0x0000000002543000 nid=0x5111 runnable [0x00007f1362bf4000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ee972c90> (a sun.nio.ch.Util$2)
        - locked <0x00000000ee972ca8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ee670000> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][[http_server_worker.default]][T#7]{New I/O worker #16}" #52 daemon prio=5 os_prio=0 tid=0x0000000002518800 nid=0x5110 runnable [0x00007f1362df5000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000eda4a150> (a sun.nio.ch.Util$2)
        - locked <0x00000000eda4a138> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ee664b08> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][[http_server_worker.default]][T#6]{New I/O worker #15}" #51 daemon prio=5 os_prio=0 tid=0x0000000002a13000 nid=0x510f runnable [0x00007f1362ff6000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ee659870> (a sun.nio.ch.Util$2)
        - locked <0x00000000ee659888> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000edebc9c8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][[http_server_worker.default]][T#5]{New I/O worker #14}" #50 daemon prio=5 os_prio=0 tid=0x0000000002aa5000 nid=0x510e runnable [0x00007f13631f7000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ee659a98> (a sun.nio.ch.Util$2)
        - locked <0x00000000ee659a80> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000edebca10> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][[http_server_worker.default]][T#4]{New I/O worker #13}" #49 daemon prio=5 os_prio=0 tid=0x0000000002a10800 nid=0x510d runnable [0x00007f13633f8000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ee671788> (a sun.nio.ch.Util$2)
        - locked <0x00000000ee671770> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ee66fee0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][[http_server_worker.default]][T#3]{New I/O worker #12}" #48 daemon prio=5 os_prio=0 tid=0x0000000002a18800 nid=0x510c runnable [0x00007f13635f9000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ede1bba8> (a sun.nio.ch.Util$2)
        - locked <0x00000000ede1bbf0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ee664b50> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][[http_server_worker.default]][T#2]{New I/O worker #11}" #47 daemon prio=5 os_prio=0 tid=0x0000000002a16800 nid=0x510b runnable [0x00007f13637fa000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ee272888> (a sun.nio.ch.Util$2)
        - locked <0x00000000ee272870> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ee66fe08> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][[http_server_worker.default]][T#1]{New I/O worker #10}" #46 daemon prio=5 os_prio=0 tid=0x0000000002a15800 nid=0x510a runnable [0x00007f13639fb000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000eddce980> (a sun.nio.ch.Util$2)
        - locked <0x00000000eddce968> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ee66fe98> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][transport_client_boss][T#1]{New I/O boss #9}" #45 daemon prio=5 os_prio=0 tid=0x00007f1378573000 nid=0x5109 runnable [0x00007f1363bfc000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000edf44350> (a sun.nio.ch.Util$2)
        - locked <0x00000000edf44338> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000edebc8f0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
        at org.elasticsearch.common.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][transport_client_worker][T#8]{New I/O worker #8}" #43 daemon prio=5 os_prio=0 tid=0x00007f1379097000 nid=0x5108 runnable [0x00007f1363dfd000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ee1d6bb0> (a sun.nio.ch.Util$2)
        - locked <0x00000000ee1d6b98> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ee664a30> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][transport_client_worker][T#7]{New I/O worker #7}" #42 daemon prio=5 os_prio=0 tid=0x00007f1379095000 nid=0x5107 runnable [0x00007f1363ffe000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000edf44140> (a sun.nio.ch.Util$2)
        - locked <0x00000000edf44128> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000edebc7d0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][transport_client_worker][T#6]{New I/O worker #6}" #41 daemon prio=5 os_prio=0 tid=0x00007f1379093800 nid=0x5106 runnable [0x00007f13683df000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ec80b538> (a sun.nio.ch.Util$2)
        - locked <0x00000000ec80b520> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ee66fd78> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][transport_client_worker][T#5]{New I/O worker #5}" #40 daemon prio=5 os_prio=0 tid=0x00007f1378e90000 nid=0x5105 runnable [0x00007f13685e0000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ee63fb18> (a sun.nio.ch.Util$2)
        - locked <0x00000000ee63fb00> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ee664958> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][transport_client_worker][T#4]{New I/O worker #4}" #39 daemon prio=5 os_prio=0 tid=0x00007f1378e8e800 nid=0x5104 runnable [0x00007f13687e1000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ee92c740> (a sun.nio.ch.Util$2)
        - locked <0x00000000ee92c728> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ee66fc58> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][transport_client_worker][T#3]{New I/O worker #3}" #38 daemon prio=5 os_prio=0 tid=0x00007f1378e7c000 nid=0x5103 runnable [0x00007f13689e2000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ee63edf8> (a sun.nio.ch.Util$2)
        - locked <0x00000000ee63ede0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000ee664838> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][transport_client_worker][T#2]{New I/O worker #2}" #37 daemon prio=5 os_prio=0 tid=0x00007f1378e7b000 nid=0x5102 runnable [0x00007f1368be3000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000edf44ab8> (a sun.nio.ch.Util$2)
        - locked <0x00000000edf44aa0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000edebcc08> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][transport_client_worker][T#1]{New I/O worker #1}" #36 daemon prio=5 os_prio=0 tid=0x00007f1378e6d800 nid=0x5101 runnable [0x00007f1368de4000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000ee689f70> (a sun.nio.ch.Util$2)
        - locked <0x00000000ee689f58> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000edebcc98> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][[ttl_expire]]" #34 daemon prio=5 os_prio=0 tid=0x00007f1378e6a800 nid=0x5100 waiting on condition [0x00007f1368fe5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ed632138> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
        at org.elasticsearch.indices.ttl.IndicesTTLService$Notifier.await(IndicesTTLService.java:325)
        at org.elasticsearch.indices.ttl.IndicesTTLService$PurgerThread.run(IndicesTTLService.java:147)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][master_mapping_updater]" #35 daemon prio=5 os_prio=0 tid=0x00007f1378e68800 nid=0x50ff waiting on condition [0x00007f13691e6000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ee6c6518> (a java.util.concurrent.LinkedTransferQueue)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:734)
        at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647)
        at java.util.concurrent.LinkedTransferQueue.poll(LinkedTransferQueue.java:1273)
        at org.elasticsearch.cluster.action.index.MappingUpdatedAction$MasterMappingUpdater.run(MappingUpdatedAction.java:382)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][scheduler][T#1]" #33 daemon prio=5 os_prio=0 tid=0x00007f137837f800 nid=0x50fe waiting on condition [0x00007f13693e7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000eb2da428> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

"elasticsearch[logstash-GKKT-SOA-BORG1-10-170-13-20-20681-11712][[timer]]" #32 daemon prio=5 os_prio=0 tid=0x00007f1378f73800 nid=0x50fd waiting on condition [0x00007f13699e8000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.elasticsearch.threadpool.ThreadPool$EstimatedTimeThread.run(ThreadPool.java:576)

"Ruby-0-Thread-18: /logstash/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0-java/lib/logstash/pipeline.rb:160" #30 daemon prio=5 os_prio=0 tid=0x00007f137858e800 nid=0x50fc in Object.wait() [0x00007f136a9f1000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:460)
        at org.jruby.RubyThread$SleepTask.run(RubyThread.java:1049)
        - locked <0x00000000ed600f00> (a org.jruby.ext.thread.SizedQueue)
        at org.jruby.RubyThread.executeBlockingTask(RubyThread.java:1065)
        at org.jruby.RubyThread.wait_timeout(RubyThread.java:1413)
        at org.jruby.ext.thread.SizedQueue.push(SizedQueue.java:133)
        - locked <0x00000000ed600f00> (a org.jruby.ext.thread.SizedQueue)
        at org.jruby.ext.thread.SizedQueue$INVOKER$i$1$0$push.call(SizedQueue$INVOKER$i$1$0$push.gen)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
        at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
        at org.jruby.runtime.Block.yield(Block.java:142)
        at org.jruby.ext.thread.Mutex.synchronize(Mutex.java:149)
        at org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
        at org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
        at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
        at org.jruby.runtime.Block.call(Block.java:101)
        at org.jruby.RubyProc.call(RubyProc.java:290)
        at org.jruby.RubyProc.call19(RubyProc.java:271)
        at org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:198)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:134)
        at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.ast.IfNode.interpret(IfNode.java:118)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.ast.WhileNode.interpret(WhileNode.java:131)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:204)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:211)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:336)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:179)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:188)
        at org.jruby.ast.CallOneArgBlockNode.interpret(CallOneArgBlockNode.java:60)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
        at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
        at org.jruby.runtime.Block.call(Block.java:101)
        at org.jruby.RubyProc.call(RubyProc.java:290)
        at org.jruby.RubyProc.call(RubyProc.java:228)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)
        at java.lang.Thread.run(Thread.java:745)

"|worker" #29 daemon prio=5 os_prio=0 tid=0x00007f137858c800 nid=0x50fb waiting on condition [0x00007f136abf2000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000edc620a0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:897)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1222)
        at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
        at org.jruby.RubyThread.lockInterruptibly(RubyThread.java:1469)
        at org.jruby.ext.thread.Mutex.lock(Mutex.java:91)
        at org.jruby.ext.thread.Mutex.synchronize(Mutex.java:147)
        at org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
        at org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.WhenOneArgNode.whenSlowTest(WhenOneArgNode.java:36)
        at org.jruby.ast.WhenOneArgNode.when(WhenOneArgNode.java:46)
        at org.jruby.ast.CaseNode.interpret(CaseNode.java:133)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.ast.WhileNode.interpret(WhileNode.java:131)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)
        at org.jruby.ast.RescueNode.interpret(RescueNode.java:116)
        at org.jruby.ast.BeginNode.interpret(BeginNode.java:83)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:139)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:187)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
        at org.jruby.ast.VCallNode.interpret(VCallNode.java:88)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
        at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
        at org.jruby.runtime.Block.call(Block.java:101)
        at org.jruby.RubyProc.call(RubyProc.java:290)
        at org.jruby.RubyProc.call(RubyProc.java:228)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)
        at java.lang.Thread.run(Thread.java:745)

"Ruby-0-Thread-2: /logstash/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.19/lib/stud/task.rb:10" #14 daemon prio=5 os_prio=0 tid=0x00007f136419d800 nid=0x50ec waiting on condition [0x00007f13710ed000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000edc620a0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:897)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1222)
        at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
        at org.jruby.RubyThread.lockInterruptibly(RubyThread.java:1469)
        at org.jruby.ext.thread.Mutex.lock(Mutex.java:91)
        at org.jruby.ext.thread.Mutex.synchronize(Mutex.java:147)
        at org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize.call(Mutex$INVOKER$i$0$0$synchronize.gen)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:316)
        at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:145)
        at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
        at org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:139)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:187)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
        at org.jruby.ast.VCallNode.interpret(VCallNode.java:88)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.ast.IfNode.interpret(IfNode.java:116)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:139)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:187)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
        at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)
        at org.jruby.ast.RescueNode.interpret(RescueNode.java:116)
        at org.jruby.ast.EnsureNode.interpret(EnsureNode.java:96)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:139)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:187)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
        at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)
        at org.jruby.ast.ReturnNode.interpret(ReturnNode.java:92)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
        at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
        at org.jruby.runtime.Block.call(Block.java:101)
        at org.jruby.RubyProc.call(RubyProc.java:290)
        at org.jruby.RubyProc.call19(RubyProc.java:271)
        at org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:198)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
        at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
        at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
        at org.jruby.runtime.Block.call(Block.java:101)
        at org.jruby.RubyProc.call(RubyProc.java:290)
        at org.jruby.RubyProc.call19(RubyProc.java:271)
        at org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:198)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
        at org.jruby.ast.CallSpecialArgNode.interpret(CallSpecialArgNode.java:65)
        at org.jruby.ast.DAsgnNode.interpret(DAsgnNode.java:110)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)
        at org.jruby.ast.RescueNode.interpret(RescueNode.java:116)
        at org.jruby.ast.BeginNode.interpret(BeginNode.java:83)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
        at org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
        at org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
        at org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
        at org.jruby.runtime.Block.call(Block.java:101)
        at org.jruby.RubyProc.call(RubyProc.java:290)
        at org.jruby.RubyProc.call(RubyProc.java:228)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)
        at java.lang.Thread.run(Thread.java:745)

"ReferenceReaper" #13 daemon prio=5 os_prio=0 tid=0x00007f1364183800 nid=0x50eb in Object.wait() [0x00007f13712f0000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        - locked <0x00000000ec8cdf20> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at org.jruby.util.ReferenceReaper$1.run(ReferenceReaper.java:32)
        at java.lang.Thread.run(Thread.java:745)

"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x0000000001d8c000 nid=0x50e6 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #8 daemon prio=9 os_prio=0 tid=0x0000000001d75800 nid=0x50e5 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #7 daemon prio=9 os_prio=0 tid=0x0000000001d73000 nid=0x50e4 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 tid=0x0000000001d71000 nid=0x50e3 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x0000000001d6f000 nid=0x50e2 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x0000000001d6d800 nid=0x50e1 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x0000000001d35000 nid=0x50e0 in Object.wait() [0x00007f1377323000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        - locked <0x00000000eb5869f0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x0000000001d32800 nid=0x50df in Object.wait() [0x00007f1377524000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
        - locked <0x00000000eb849668> (a java.lang.ref.Reference$Lock)

"LogStash::Runner" #1 prio=5 os_prio=0 tid=0x0000000001c2d000 nid=0x50d8 in Object.wait() [0x00007f138f3a7000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Thread.join(Thread.java:1253)
        - locked <0x00000000ebd455e8> (a java.lang.Thread)
        at org.jruby.internal.runtime.NativeThread.join(NativeThread.java:75)
        at org.jruby.RubyThread.join(RubyThread.java:696)
        at org.jruby.RubyThread$INVOKER$i$0$1$join.call(RubyThread$INVOKER$i$0$1$join.gen)
        at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:661)
        at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:198)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
        at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:139)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:187)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:306)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:136)
        at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:60)
        at org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)
        at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
        at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
        at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
        at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)
        at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)
        at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)
        at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:170)
        at logstash.logstash.lib.bootstrap.environment.__file__(/logstash/logstash/lib/bootstrap/environment.rb:48)
        at logstash.logstash.lib.bootstrap.environment.load(/logstash/logstash/lib/bootstrap/environment.rb)
        at org.jruby.Ruby.runScript(Ruby.java:866)
        at org.jruby.Ruby.runScript(Ruby.java:859)
        at org.jruby.Ruby.runNormally(Ruby.java:728)
        at org.jruby.Ruby.runFromMain(Ruby.java:577)
        at org.jruby.Main.doRunFromMain(Main.java:395)
        at org.jruby.Main.internalRun(Main.java:290)
        at org.jruby.Main.run(Main.java:217)
        at org.jruby.Main.main(Main.java:197)

"VM Thread" os_prio=0 tid=0x0000000001d2d800 nid=0x50de runnable

"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x0000000001c3e000 nid=0x50d9 runnable

"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x0000000001c40000 nid=0x50da runnable

"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x0000000001c41800 nid=0x50db runnable

"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x0000000001c43000 nid=0x50dc runnable

"Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x0000000001c86000 nid=0x50dd runnable

"VM Periodic Task Thread" os_prio=0 tid=0x0000000001d8f000 nid=0x50e7 waiting on condition

JNI global references: 272

Heap
 par new generation   total 38720K, used 26415K [0x00000000e0c00000, 0x00000000e3600000, 0x00000000eb2a0000)
  eden space 34432K,  76% used [0x00000000e0c00000, 0x00000000e25c9d68, 0x00000000e2da0000)
  from space 4288K,   0% used [0x00000000e31d0000, 0x00000000e31d2040, 0x00000000e3600000)
  to   space 4288K,   0% used [0x00000000e2da0000, 0x00000000e2da0000, 0x00000000e31d0000)
 concurrent mark-sweep generation total 206432K, used 92759K [0x00000000eb2a0000, 0x00000000f7c38000, 0x0000000100000000)
 Metaspace       used 57404K, capacity 60158K, committed 60408K, reserved 1099776K
  class space    used 8634K, capacity 9255K, committed 9380K, reserved 1048576K
Centopus commented 9 years ago

I'm thinking of a way, to anonymize the data from the log file. As I look at it now, I'm not allowed to send it to you. (customer information etc...) I cant find anything of interesting in there anyway.... (any clues what should I be looking for, to help?)....

Working on anonymizing of the data... but looking at it, the messages just stop coming. No error/indication of cause. I've copied the 'interesting' time segments on the end of this post. (but there is nothing there...)

http://pasteboard.co/IQPgtvY.png This is how the data stream looked before it hung on me. It should look like that part before 11:15.

Heapdump (22MB, tgz file): https://onedrive.live.com/redir?resid=C4D6FEDB5CC8F8B5!120&authkey=!AEVOys9qgzXPsiY&ithint=file%2ctgz

First gap:

{:timestamp=>"2015-05-25T11:14:54.690000+0200", :level=>:warn, "message"=>"10.170.7.135 - - [25/May/2015:11:14:53 +0200] \"POST /contractor/ContractorRegistrySecuredExtGateway HTTP/1.1\" 200 914 BNqej1MI500000000 gkkt-osb5-vip-priv:8411", "@version"=>"1", "@timestamp"=>"2015-05-25T09:14:54.677Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd1"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-1/net-TPE-PROD/logs/TPE-PROD-OSB-access-noSSL.log"}
{:timestamp=>"2015-05-25T11:14:54.690000+0200", :level=>:warn, "message"=>"10.170.7.135 - - [25/May/2015:11:14:53 +0200] \"POST /contractor/ContractorRegistrySecuredExtGateway HTTP/1.1\" 200 3333 7q9Qe0Na500000000 gkkt-osb1-vip-priv:8411", "@version"=>"1", "@timestamp"=>"2015-05-25T09:14:54.677Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd1"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-1/net-TPE-PROD/logs/TPE-PROD-OSB-access-noSSL.log"}
{:timestamp=>"2015-05-25T11:14:54.690000+0200", :level=>:warn, "message"=>"10.240.254.29 - - [25/May/2015:11:14:53 +0200] \"POST /contractor/ContractorRegistrySecuredExtGateway HTTP/1.1\" 200 3289 Yvmzg1QQ000000000 gkkt-osb5-vip-priv:8411", "@version"=>"1", "@timestamp"=>"2015-05-25T09:14:54.678Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd1"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-1/net-TPE-PROD/logs/TPE-PROD-OSB-access-noSSL.log"}
{:timestamp=>"2015-05-25T11:14:54.690000+0200", :level=>:warn, "message"=>"10.170.12.147 - - [25/May/2015:11:14:53 +0200] \"POST /Invasion/InvasionGateway HTTP/1.1\" 200 193 BNqej1Du500000000 gkkt-osb4-vip-priv:8411", "@version"=>"1", "@timestamp"=>"2015-05-25T09:14:54.678Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd1"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-1/net-TPE-PROD/logs/TPE-PROD-OSB-access-noSSL.log"}
{:timestamp=>"2015-05-25T11:14:55.391000+0200", :level=>:warn, "message"=>"192.168.1.2 - - [25/May/2015:11:14:54 +0200] \"POST /soa-infra/services/default/BorgSoaContractorRegistryPrj/ContractorRegistryExtProcMediator_ep?WSDL HTTP/1.1\" 200 5488 3sFor14p100000000 gkkt-soa2-vip-priv:8401", "@version"=>"1", "@timestamp"=>"2015-05-25T09:14:55.388Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd2"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-2/net-TPE-PROD/logs/TPE-PROD-SOA-access.log"}
{:timestamp=>"2015-05-25T11:14:55.391000+0200", :level=>:warn, "message"=>"192.168.1.6 - - [25/May/2015:11:14:54 +0200] \"POST /soa-infra/services/default/BorgSoaGosiaPrj/GosiaCityMediator_ep HTTP/1.1\" 200 15805 M^lZg1BN500000000 gkkt-soa1-vip-priv:8401", "@version"=>"1", "@timestamp"=>"2015-05-25T09:14:55.388Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd2"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-2/net-TPE-PROD/logs/TPE-PROD-SOA-access.log"}
{:timestamp=>"2015-05-25T11:14:55.693000+0200", :level=>:warn, "message"=>"10.170.4.75 - - [25/May/2015:11:14:54 +0200] \"POST /Invasion/InvasionGateway HTTP/1.1\" 200 2065 kaicu09N000000000 gkkt-osb3-vip-priv:8411", "@version"=>"1", "@timestamp"=>"2015-05-25T09:14:55.690Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd1"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-1/net-TPE-PROD/logs/TPE-PROD-OSB-access-noSSL.log"}
{:timestamp=>"2015-05-25T11:14:55.694000+0200", :level=>:warn, "message"=>"10.170.7.134 - - [25/May/2015:11:14:54 +0200] \"POST /gosia/CityGateway HTTP/1.1\" 200 15805 BNqej12c500000000 gkkt-osb4-vip-priv:8411", "@version"=>"1", "@timestamp"=>"2015-05-25T09:14:55.691Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd1"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-1/net-TPE-PROD/logs/TPE-PROD-OSB-access-noSSL.log"}
{:timestamp=>"2015-05-25T11:14:55.694000+0200", :level=>:warn, "message"=>"10.240.254.29 - - [25/May/2015:11:14:54 +0200] \"POST /contractor/ContractorRegistrySecuredExtGateway HTTP/1.1\" 200 5527 Yvmzg1RQ000000000 gkkt-osb2-vip-priv:8411", "@version"=>"1", "@timestamp"=>"2015-05-25T09:14:55.691Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd1"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-1/net-TPE-PROD/logs/TPE-PROD-OSB-access-noSSL.log"}
{:timestamp=>"2015-05-25T11:18:03.970000+0200", :level=>:warn, "message"=>"####<May 25, 2015 11:13:55 AM CEST> <Error> <Cluster> <GKKT-SOA-TEST6> <osb_server6> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <2c69563da95f74ab:5ed644d2:14d8548681a:-8000-000000000001abcc> <1432545235595> <BEA-000168> <Failed to restart/migrate server \"osb_server33\" because of Failed to start the migratable server on one of the candidate machines> ", "@version"=>"1", "@timestamp"=>"2015-05-25T09:16:10.594Z", "type"=>"WLlog", "tags"=>["TEST", "Domain", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/AdminServer/TPE-SOA-TEST.log"}
{:timestamp=>"2015-05-25T11:18:03.970000+0200", :level=>:warn, "message"=>"####<May 25, 2015 11:14:05 AM CEST> <Error> <Cluster> <GKKT-SOA-TEST6> <osb_server6> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <2c69563da95f74ab:5ed644d2:14d8548681a:-8000-000000000001abd9> <1432545245596> <BEA-000168> <Failed to restart/migrate server \"osb_server33\" because of Failed to start the migratable server on one of the candidate machines> ", "@version"=>"1", "@timestamp"=>"2015-05-25T09:18:03.968Z", "type"=>"WLlog", "tags"=>["TEST", "Domain", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/AdminServer/TPE-SOA-TEST.log"}
{:timestamp=>"2015-05-25T11:18:03.970000+0200", :level=>:warn, "message"=>"####<May 25, 2015 11:14:15 AM CEST> <Error> <Cluster> <GKKT-SOA-TEST6> <osb_server6> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <2c69563da95f74ab:5ed644d2:14d8548681a:-8000-000000000001abe4> <1432545255598> <BEA-000168> <Failed to restart/migrate server \"osb_server33\" because of Failed to start the migratable server on one of the candidate machines> ", "@version"=>"1", "@timestamp"=>"2015-05-25T09:18:03.968Z", "type"=>"WLlog", "tags"=>["TEST", "Domain", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/AdminServer/TPE-SOA-TEST.log"}

Second (large) gap:

{:timestamp=>"2015-05-25T11:19:28.598000+0200", :level=>:warn, "message"=>"192.168.1.5 - - [25/May/2015:11:19:27 +0200] \"POST /soa-infra/services/default/RoutePrj/routeprocess_client_ep HTTP/1.1\" 200 959 kM9QK1h8500000000 gkkt-soa3-vip-priv:8401", "@version"=>"1", "@timestamp"=>"2015-05-25T09:19:28.595Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd2"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-2/net-TPE-PROD/logs/TPE-PROD-SOA-access.log"}
{:timestamp=>"2015-05-25T11:19:28.888000+0200", :level=>:warn, "message"=>"10.170.9.54 - - [25/May/2015:11:19:27 +0200] \"POST /Invasion/InvasionGateway HTTP/1.1\" 200 998 vrSN^0oo600000000 gkkt-osb3-vip-priv:8411", "@version"=>"1", "@timestamp"=>"2015-05-25T09:19:28.704Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd1"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-1/net-TPE-PROD/logs/TPE-PROD-OSB-access-noSSL.log"}
{:timestamp=>"2015-05-25T11:19:28.889000+0200", :level=>:warn, "message"=>"10.170.13.7 - - [25/May/2015:11:19:27 +0200] \"POST /Invasion/InvasionGateway HTTP/1.1\" 200 545 kM9QK1h8500000000 gkkt-osb5-vip-priv:8411", "@version"=>"1", "@timestamp"=>"2015-05-25T09:19:28.885Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd1"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-1/net-TPE-PROD/logs/TPE-PROD-OSB-access-noSSL.log"}
{:timestamp=>"2015-05-25T11:19:28.889000+0200", :level=>:warn, "message"=>"10.170.13.6 - - [25/May/2015:11:19:27 +0200] \"POST /Invasion/InvasionGateway HTTP/1.1\" 200 747 GKzph1lY500000000 gkkt-osb1-vip-priv:8411", "@version"=>"1", "@timestamp"=>"2015-05-25T09:19:28.885Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd1"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-1/net-TPE-PROD/logs/TPE-PROD-OSB-access-noSSL.log"}
{:timestamp=>"2015-05-25T11:19:28.897000+0200", :level=>:warn, "message"=>"[2015-05-25T11:19:28+02:00] [net-TPE-PROD] [WARNING:1] [OTD-11016] [] [pid: 8390] health-check reports: all servers in origin server pool PROD-BAM-NSDP are offline.", "@version"=>"1", "@timestamp"=>"2015-05-25T09:19:28.887Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd1"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-1/net-TPE-PROD/logs/server.log"}
{:timestamp=>"2015-05-25T11:19:29.631000+0200", :level=>:warn, "message"=>"192.168.1.6 - - [25/May/2015:11:19:28 +0200] \"POST /soa-infra/services/default/BorgSoaContractorRegistryPrj/ContractorRegistryExtProcMediator_ep?WSDL HTTP/1.1\" 200 3321 xFaui0Oa500000000 gkkt-soa1-vip-priv:8401", "@version"=>"1", "@timestamp"=>"2015-05-25T09:19:29.600Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd2"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-2/net-TPE-PROD/logs/TPE-PROD-SOA-access.log"}
{:timestamp=>"2015-05-25T11:19:29.900000+0200", :level=>:warn, "message"=>"10.170.7.134 - - [25/May/2015:11:19:28 +0200] \"POST /contractor/ContractorRegistrySecuredExtGateway HTTP/1.1\" 200 3360 aWUke1ZXG00000000 gkkt-osb4-vip-priv:8411", "@version"=>"1", "@timestamp"=>"2015-05-25T09:19:29.897Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd1"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-1/net-TPE-PROD/logs/TPE-PROD-OSB-access-noSSL.log"}
{:timestamp=>"2015-05-25T11:19:29.901000+0200", :level=>:warn, "message"=>"10.170.6.146 - - [25/May/2015:11:19:28 +0200] \"POST /CARD/ClosePurchaseContractGateway HTTP/1.1\" 200 0 GmNtq0EY500000000 gkkt-osb3-vip-priv:8411", "@version"=>"1", "@timestamp"=>"2015-05-25T09:19:29.898Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd1"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-1/net-TPE-PROD/logs/TPE-PROD-OSB-access-noSSL.log"}
{:timestamp=>"2015-05-25T11:19:30.682000+0200", :level=>:warn, "message"=>"192.168.1.7 - - [25/May/2015:11:19:29 +0200] \"POST /soa-infra/services/default/BorgSoaContractorRegistryPrj/ContractorRegistryExtProcMediator_ep?WSDL HTTP/1.1\" 200 3303 n9ViV1Pu500000000 gkkt-soa6-vip-priv:8401", "@version"=>"1", "@timestamp"=>"2015-05-25T09:19:30.632Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd2"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-2/net-TPE-PROD/logs/TPE-PROD-SOA-access.log"}
{:timestamp=>"2015-05-25T11:19:30.953000+0200", :level=>:warn, "message"=>"10.1.27.167 - - [25/May/2015:11:19:29 +0200] \"POST /Invasion/InvasionGateway HTTP/1.1\" 200 775 9vz0V1El600000000 gkkt-osb4-vip-priv:8411", "@version"=>"1", "@timestamp"=>"2015-05-25T09:19:30.901Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd1"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-1/net-TPE-PROD/logs/TPE-PROD-OSB-access-noSSL.log"}
{:timestamp=>"2015-05-25T11:19:30.954000+0200", :level=>:warn, "message"=>"10.170.4.68 - - [25/May/2015:11:19:29 +0200] \"POST /Invasion/InvasionGateway HTTP/1.1\" 200 602 Ns7tH1ns400000000 gkkt-osb5-vip-priv:8411", "@version"=>"1", "@timestamp"=>"2015-05-25T09:19:30.951Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd1"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-1/net-TPE-PROD/logs/TPE-PROD-OSB-access-noSSL.log"}
{:timestamp=>"2015-05-25T11:19:30.955000+0200", :level=>:warn, "message"=>"10.240.254.29 - - [25/May/2015:11:19:29 +0200] \"POST /contractor/ContractorRegistrySecuredExtGateway HTTP/1.1\" 200 3342 DprBA0ZR000000000 gkkt-osb5-vip-priv:8411", "@version"=>"1", "@timestamp"=>"2015-05-25T09:19:30.951Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd1"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-1/net-TPE-PROD/logs/TPE-PROD-OSB-access-noSSL.log"}
{:timestamp=>"2015-05-25T11:19:30.955000+0200", :level=>:warn, "message"=>"10.170.4.68 - - [25/May/2015:11:19:29 +0200] \"POST /Invasion/InvasionGateway HTTP/1.1\" 200 578 Ns7tH1os400000000 gkkt-osb1-vip-priv:8411", "@version"=>"1", "@timestamp"=>"2015-05-25T09:19:30.951Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd1"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-1/net-TPE-PROD/logs/TPE-PROD-OSB-access-noSSL.log"}
{:timestamp=>"2015-05-25T11:25:12.335000+0200", :level=>:warn, "message"=>"####<May 25, 2015 11:17:15 AM CEST> <Error> <Cluster> <GKKT-SOA-TEST6> <osb_server6> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <2c69563da95f74ab:5ed644d2:14d8548681a:-8000-000000000001acae> <1432545435625> <BEA-000168> <Failed to restart/migrate server \"osb_server33\" because of Failed to start the migratable server on one of the candidate machines> ", "@version"=>"1", "@timestamp"=>"2015-05-25T09:19:44.467Z", "type"=>"WLlog", "tags"=>["TEST", "Domain", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/AdminServer/TPE-SOA-TEST.log"}
{:timestamp=>"2015-05-25T11:25:12.335000+0200", :level=>:warn, "message"=>"####<May 25, 2015 11:17:25 AM CEST> <Error> <Cluster> <GKKT-SOA-TEST6> <osb_server6> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <2c69563da95f74ab:5ed644d2:14d8548681a:-8000-000000000001acb7> <1432545445626> <BEA-000168> <Failed to restart/migrate server \"osb_server33\" because of Failed to start the migratable server on one of the candidate machines> ", "@version"=>"1", "@timestamp"=>"2015-05-25T09:25:12.329Z", "type"=>"WLlog", "tags"=>["TEST", "Domain", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/AdminServer/TPE-SOA-TEST.log"}
{:timestamp=>"2015-05-25T11:25:12.336000+0200", :level=>:warn, "message"=>"10.170.13.235 - - [25/May/2015:11:19:32 +0200] \"OPTIONS / HTTP/1.1\" 404 1164 j3qii0tR000000000 gkkt-osb5-vip-test:8411", "@version"=>"1", "@timestamp"=>"2015-05-25T09:19:34.586Z", "type"=>"TRlog", "tags"=>["TEST", "traf", "log", "vtd1"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TPE-OSB-TEST-access.log"}
{:timestamp=>"2015-05-25T11:25:12.337000+0200", :level=>:warn, "message"=>"10.170.13.215 - - [25/May/2015:11:19:47 +0200] \"OPTIONS / HTTP/1.1\" 404 1164 PbDOK0PL000000000 gkkt-osb6-vip-test:8411", "@version"=>"1", "@timestamp"=>"2015-05-25T09:25:12.332Z", "type"=>"TRlog", "tags"=>["TEST", "traf", "log", "vtd1"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/otdcfg/otd-inst-1/net-TPE-TEST/logs/TPE-OSB-TEST-access.log"}

The end:

{:timestamp=>"2015-05-25T11:26:06.902000+0200", :level=>:warn, "message"=>"####<May 25, 2015 11:25:45 AM CEST> <Error> <Cluster> <GKKT-SOA-TEST6> <osb_server6> <[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <2c69563da95f74ab:5ed644d2:14d8548681a:-8000-000000000001aedd> <1432545945697> <BEA-000168> <Failed to restart/migrate server \"osb_server33\" because of Failed to start the migratable server on one of the candidate machines> ", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.901Z", "type"=>"WLlog", "tags"=>["TEST", "OSB", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/osb_server6/osb_server6.log"}
{:timestamp=>"2015-05-25T11:26:06.903000+0200", :level=>:warn, "message"=>"####<May 25, 2015 11:24:25 AM CEST> <Error> <Cluster> <GKKT-SOA-TEST6> <osb_server6> <[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <2c69563da95f74ab:5ed644d2:14d8548681a:-8000-000000000001ae83> <1432545865685> <BEA-000168> <Failed to restart/migrate server \"osb_server33\" because of Failed to start the migratable server on one of the candidate machines> ", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.901Z", "type"=>"WLlog", "tags"=>["TEST", "Domain", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/AdminServer/TPE-SOA-TEST.log"}
{:timestamp=>"2015-05-25T11:26:06.903000+0200", :level=>:warn, "message"=>"####<May 25, 2015 11:24:35 AM CEST> <Error> <Cluster> <GKKT-SOA-TEST6> <osb_server6> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <2c69563da95f74ab:5ed644d2:14d8548681a:-8000-000000000001ae8e> <1432545875686> <BEA-000168> <Failed to restart/migrate server \"osb_server33\" because of Failed to start the migratable server on one of the candidate machines> ", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.901Z", "type"=>"WLlog", "tags"=>["TEST", "Domain", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/AdminServer/TPE-SOA-TEST.log"}
{:timestamp=>"2015-05-25T11:26:06.903000+0200", :level=>:warn, "message"=>" [OSB Tracing] Inbound request was received. ", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.901Z", "type"=>"WLlog", "tags"=>["TEST", "OSB", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/osb_server6/osb_server6.log"}
{:timestamp=>"2015-05-25T11:26:06.903000+0200", :level=>:warn, "message"=>"####<May 25, 2015 11:24:45 AM CEST> <Error> <Cluster> <GKKT-SOA-TEST6> <osb_server6> <[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <2c69563da95f74ab:5ed644d2:14d8548681a:-8000-000000000001ae99> <1432545885688> <BEA-000168> <Failed to restart/migrate server \"osb_server33\" because of Failed to start the migratable server on one of the candidate machines> ", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.901Z", "type"=>"WLlog", "tags"=>["TEST", "Domain", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/AdminServer/TPE-SOA-TEST.log"}
{:timestamp=>"2015-05-25T11:26:06.904000+0200", :level=>:warn, "message"=>" ", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.901Z", "type"=>"WLlog", "tags"=>["TEST", "OSB", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/osb_server6/osb_server6.log"}
{:timestamp=>"2015-05-25T11:26:06.904000+0200", :level=>:warn, "message"=>"####<May 25, 2015 11:24:55 AM CEST> <Error> <Cluster> <GKKT-SOA-TEST6> <osb_server6> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <2c69563da95f74ab:5ed644d2:14d8548681a:-8000-000000000001aea2> <1432545895689> <BEA-000168> <Failed to restart/migrate server \"osb_server33\" because of Failed to start the migratable server on one of the candidate machines> ", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.901Z", "type"=>"WLlog", "tags"=>["TEST", "Domain", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/AdminServer/TPE-SOA-TEST.log"}
{:timestamp=>"2015-05-25T11:26:06.904000+0200", :level=>:warn, "message"=>" Service Ref = CARD/ProxyService/NikeBlackboardGateway", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.901Z", "type"=>"WLlog", "tags"=>["TEST", "OSB", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/osb_server6/osb_server6.log"}
{:timestamp=>"2015-05-25T11:26:06.904000+0200", :level=>:warn, "message"=>"####<May 25, 2015 11:25:05 AM CEST> <Error> <Cluster> <GKKT-SOA-TEST6> <osb_server6> <[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <2c69563da95f74ab:5ed644d2:14d8548681a:-8000-000000000001aead> <1432545905691> <BEA-000168> <Failed to restart/migrate server \"osb_server33\" because of Failed to start the migratable server on one of the candidate machines> ", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.901Z", "type"=>"WLlog", "tags"=>["TEST", "Domain", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/AdminServer/TPE-SOA-TEST.log"}
{:timestamp=>"2015-05-25T11:26:06.904000+0200", :level=>:warn, "message"=>" URI = /CARD/NikeBlackboardGateway", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.901Z", "type"=>"WLlog", "tags"=>["TEST", "OSB", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/osb_server6/osb_server6.log"}
{:timestamp=>"2015-05-25T11:26:06.905000+0200", :level=>:warn, "message"=>" Message ID = 4831425868103297363-5ed644d2.14d8548681a.-7f59", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.901Z", "type"=>"WLlog", "tags"=>["TEST", "OSB", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/osb_server6/osb_server6.log"}
{:timestamp=>"2015-05-25T11:26:06.905000+0200", :level=>:warn, "message"=>"####<May 25, 2015 11:25:15 AM CEST> <Error> <Cluster> <GKKT-SOA-TEST6> <osb_server6> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <2c69563da95f74ab:5ed644d2:14d8548681a:-8000-000000000001aebe> <1432545915692> <BEA-000168> <Failed to restart/migrate server \"osb_server33\" because of Failed to start the migratable server on one of the candidate machines> ", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.901Z", "type"=>"WLlog", "tags"=>["TEST", "Domain", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/AdminServer/TPE-SOA-TEST.log"}
{:timestamp=>"2015-05-25T11:26:06.905000+0200", :level=>:warn, "message"=>" Request metadata =", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.901Z", "type"=>"WLlog", "tags"=>["TEST", "OSB", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/osb_server6/osb_server6.log"}
{:timestamp=>"2015-05-25T11:26:06.905000+0200", :level=>:warn, "message"=>"####<May 25, 2015 11:25:25 AM CEST> <Error> <Cluster> <GKKT-SOA-TEST6> <osb_server6> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <2c69563da95f74ab:5ed644d2:14d8548681a:-8000-000000000001aec7> <1432545925694> <BEA-000168> <Failed to restart/migrate server \"osb_server33\" because of Failed to start the migratable server on one of the candidate machines> ", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.901Z", "type"=>"WLlog", "tags"=>["TEST", "Domain", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/AdminServer/TPE-SOA-TEST.log"}
{:timestamp=>"2015-05-25T11:26:06.906000+0200", :level=>:warn, "message"=>"    <xml-fragment>", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.901Z", "type"=>"WLlog", "tags"=>["TEST", "OSB", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/osb_server6/osb_server6.log"}
{:timestamp=>"2015-05-25T11:26:06.906000+0200", :level=>:warn, "message"=>"        <http:Accept>*/*</http:Accept>", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.902Z", "type"=>"WLlog", "tags"=>["TEST", "OSB", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/osb_server6/osb_server6.log"}
{:timestamp=>"2015-05-25T11:26:06.906000+0200", :level=>:warn, "message"=>"        <http:Connection>keep-alive</http:Connection>", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.902Z", "type"=>"WLlog", "tags"=>["TEST", "OSB", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/osb_server6/osb_server6.log"}
{:timestamp=>"2015-05-25T11:26:06.907000+0200", :level=>:warn, "message"=>"        <http:Content-Length>709</http:Content-Length>", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.902Z", "type"=>"WLlog", "tags"=>["TEST", "OSB", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/osb_server6/osb_server6.log"}
{:timestamp=>"2015-05-25T11:26:06.907000+0200", :level=>:warn, "message"=>"192.168.1.1 - - [25/May/2015:11:25:46 +0200] \"POST /soa-infra/services/default/BorgSoaGosiaPrj/GosiaStreetMediator_ep HTTP/1.1\" 200 1238 bwN0P1Si500000000 gkkt-soa6-vip-priv:8401", "@version"=>"1", "@timestamp"=>"2015-05-25T09:25:50.142Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd2"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-2/net-TPE-PROD/logs/TPE-PROD-SOA-access.log"}
{:timestamp=>"2015-05-25T11:26:06.913000+0200", :level=>:warn, "message"=>"192.168.1.6 - - [25/May/2015:11:25:47 +0200] \"POST /soa-infra/services/default/BorgSoaGosiaPrj/GosiaStreetMediator_ep HTTP/1.1\" 200 1238 vb^d11Ze500000000 gkkt-soa5-vip-priv:8401", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.904Z", "type"=>"TRlog", "tags"=>["PROD", "traf", "log", "vtd2"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/PROD/otdcfg/otd-inst-2/net-TPE-PROD/logs/TPE-PROD-SOA-access.log"}
{:timestamp=>"2015-05-25T11:26:06.913000+0200", :level=>:warn, "message"=>"        <http:Content-Type>text/xml; charset=utf-8</http:Content-Type>", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.904Z", "type"=>"WLlog", "tags"=>["TEST", "OSB", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/osb_server6/osb_server6.log"}
{:timestamp=>"2015-05-25T11:26:06.915000+0200", :level=>:warn, "message"=>"        <http:Host>testsoabalancer.tpe.corp:8011</http:Host>", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.910Z", "type"=>"WLlog", "tags"=>["TEST", "OSB", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/osb_server6/osb_server6.log"}
{:timestamp=>"2015-05-25T11:26:06.916000+0200", :level=>:warn, "message"=>"        <http:SOAPAction/>", "@version"=>"1", "@timestamp"=>"2015-05-25T09:26:06.913Z", "type"=>"WLlog", "tags"=>["TEST", "OSB", "log"], "host"=>"GKKT-SOA-BORG1-10-170-13-20", "path"=>"/TEST/logs/osb_server6/osb_server6.log"

I'm running logstash now, with reduced number of logfiles to parse... no sensitive data in there. I hope it will hang, I'll be able to post whole log file then.

ph commented 9 years ago

@Centopus OK, by looking at your thread dump I think this isn't the same issue as #3302.

The lecture of the thread dump show that the filters and the outputs are blocked in a waiting state (waiting for new events). The dump doesn't show any trace of the file input threads they should be named something like "<file", this is concerning because the pipeline usually warn users if a thread dies.

@Centopus there is no trace of exception in the logstash logfile at all?

ph commented 9 years ago

@parabolic I've look again with your thread dump, they show a healthy logstash instance to me. The thread dump is a bit more useful when logstash is stuck/hang because we will know on what part of the code logstash is stopped so this may give us information on the problematic plugins.

ph commented 9 years ago

@parabolic I have just noted that you are also exporting all your events as file! This is really awesome!

Can you try to reproduce the issue with the physical file on a separate logstash 1.5 environment? If we can narrow it down to a specific reproducible pattern this would be really helpful to me!

Centopus commented 9 years ago

@ph Yes, no trace of exception or error. As I mentioned in my discuss.elastic.co post I've ran logstash with --debug flag, and noticed the reading threads dissappear/end - no error to be seen. At some moment they are there, and after a while they stop reporting.

Do you have an idea how to monitor one of those? To see if the thread itself can provide us with any data before it ends.

Unfortunately the test without the sensitive data is still going strong, not looking like its going to hang. I've removed this from my config: 

file {
        type => "WLlog"
        path => ["/TEST/logs/AdminServer/AdminServer.log"]
        tags => [TEST,Admin,log]
       }
  file {
        type => "WLlog"
        path => ["/PROD/logs/AdminServer/AdminServer.log"]
        tags => [PROD,Admin,log]
       }
  file {
        type => "WLlog"
        path => ["/TEST/logs/AdminServer/TEST.log"]
        tags => [TEST,Domain,log]
        }
  file {
        type => "WLlog"
        path => ["/PROD/logs/AdminServer/PROD.log"]
        tags => [PROD,Domain,log]
        }
  file {
        type => "WLlog"
        path => [
                        "/TEST/logs/osb_server1/osb_server1.log",
                        "/TEST/logs/osb_server2/osb_server2.log",
                        "/TEST/logs/osb_server3/osb_server3.log",
                        "/TEST/logs/osb_server4/osb_server4.log",
                        "/TEST/logs/osb_server5/osb_server5.log",
                        "/TEST/logs/osb_server6/osb_server6.log"
                ]
        tags => [TEST,OSB,log]
        }
 file {
        type => "WLlog"
        path => [
                        "/PROD/logs/osb_server1/osb_server1.log",
                        "/PROD/logs/osb_server2/osb_server2.log",
                        "/PROD/logs/osb_server3/osb_server3.log",
                        "/PROD/logs/osb_server4/osb_server4.log",
                        "/PROD/logs/osb_server5/osb_server5.log",
                        "/PROD/logs/osb_server6/osb_server6.log"
                ]
        tags => [PROD,OSB,log]
        }
  file {
        type => "WLlog"
        path => [
                        "/TEST/logs/soa_server1/soa_server1.log",
                        "/TEST/logs/soa_server2/soa_server2.log",
                        "/TEST/logs/soa_server3/soa_server3.log",
                        "/TEST/logs/soa_server4/soa_server4.log",
                        "/TEST/logs/soa_server5/soa_server5.log",
                        "/TEST/logs/soa_server6/soa_server6.log"
                ]
        tags => [TEST,SOA,log]
        }
  file {
        type => "WLlog"
        path => [
                        "/PROD/logs/soa_server1/soa_server1.log",
                        "/PROD/logs/soa_server2/soa_server2.log",
                        "/PROD/logs/soa_server3/soa_server3.log",
                        "/PROD/logs/soa_server4/soa_server4.log",
                        "/PROD/logs/soa_server5/soa_server5.log",
                        "/PROD/logs/soa_server6/soa_server6.log"
                ]
        tags => [PROD,SOA,log]
        }

I'm starting to think, that those 'six-packs' might be the cause. I'm going to wait until the morning rush is over (i'm still hoping it hangs) and then try to add this back, but separate file inputs for each file, no bundling.

Morning rush didn't hang it. (Sadly, the volume of data without weblogics is 1/20 of what comes from there...)

Separating the 'six-packs' into separate file{} inputs didn't help, still getting hangups.

Adding now syslog input, to see whether only file inputs die this way, or all inputs... will report when I have more information.

I got it hung with additional input (syslog -> tcp input plugin). 

tcp {
       port => 514
       type => "syslog"
      }
  udp {
       port => 514
       type => "syslog"
      }

While hung, 'syslog' does not take it any data too... Threaddump: http://pastebin.com/mpGRgH4u Config now is same as initial one, just everything separated into separate input plugins (file), plus tcp input, plus some uncomplicated filters for the syslog stuff (with translate plugin ).

ph commented 9 years ago

@parabolic You could use a globbing paths for the file inputs, like this.

 file {
        type => "WLlog"
        path => "/logs/soa_server1/soa_server*.log",
}

This will also watch for new file in the directory.

Centopus commented 9 years ago

@ph I do believe you wanted to talk to me, when talking about globbing paths. Now I'm unsure who were you referring to, in your previous comments.

There are many files in those folders, and I just want to read single one out of all those folders. I'd need to do it like that: "/logs/_server/*.log" to pick out the files I need to be read. I do think its irrelevant to the problems I'm noticing.

I'm doing now a test run with logstash 1.4.0 + elasticsearch 1.5.0 on the same machine. (had to add protocol => http to the output config, because of the version difference) to see if it will happen too.

Two more things I'm thinking about: -> trying to run logstash 1.5.0 on a completely different machine to see if the problem is not bound to my test environment.
-> older java version

ph commented 9 years ago

@Centopus Awesome, keep us in the loop we need to narrow this issue down!

Centopus commented 9 years ago

Update: logstash 1.4.0 (protocol=>http) + elastic 1.5.1 worked fine for over 20h, on my test environment, where 1.5,0 + 1.5.1 failed; assuming this is ok and stable. (both logstash and elastic on same machine)

Running now logstash 1.5.0 (protocol => default = node, there is no such line in config file now) on completely different machine, connecting to the same elasticsearch 1.5.1, that is on my initial machine. , using same java version (jdk1.8.0_45), no tcp/syslog input (irrelevant, it hung itself without that too).

Ah, one more thing that might be relevant. My whole logstash+elasticsearch install is on a NFS share, as are all log files I'm reading. I haven't mentioned that earlier, because my 'old' production setup (LS1.4.0 + ES1.1.1) worked fine with that for at least 6 months now, maybe more.

duylong commented 9 years ago

It seems, we have the same problem.

I use multiple rabbitmq (input and output).

After a massive amounts of logs, Logstash can block and we must restart the service. No errors in the log file and no problem with Logstash 1.4.0.

Java version : 1.7.0.79

Centopus commented 9 years ago

LS 1.5.0 on separate machine, pushing data to ES 1.5.1 just died... ;> therefore it is not specific to my machine I've started on. No errors or anything, just stops putting data into ES.

Trying now 1.5.0 with (protocol=>http) //I'm not a fan... i like seeing my logstash instances from kopf.. ^^ died already.

Trying now with java jdk-7u79. (protocol=> node , as it has been proven, it does not influence the problem).

jsvd commented 9 years ago

@Centopus so, so far you have

logstash 1.4 file => elasticsearch (http protocol) OK

logstash 1.5 file => elasticsearch (node protocol) HANG file + tcp => elasticsearch (node protocol) HANG file + tcp => elasticsearch (http protocol) ???

is that correct?

Centopus commented 9 years ago

@jsvd logstash 1.4 file + tcp => ES 1.5 (http) OK // LS+ES on the same machine, jdk1.8.0_45 file + tcp => ES 1.1 (node) OK // LS+ES on the same machine, jdk1.6.0_45 (this is my current stable production setup working all the time on separate server)

logstash 1.5 file = > ES1.5 (node) hang // LS+ES same machine, jdk1.8.0_45 file = > ES1.5 (node) hang // LS+ES 2 separate machines, jdk1.8.0_45 file + tcp = > ES1.5 (node) hang // LS+ES on the same machine, jdk1.8.0_45 file + tcp = > ES1.5 (http) hang // LS+ES on the same machine, jdk1.8.0_45

Now I'm going to try: logstash 1.5 file + tcp = > ES1.5 (node) // LS+ES on the same machine, jdk1.7.0_79

I'm running out of ideas...

I think that the output protocol choice does not change anything. Also my volume on tcp input is too low, to check if it alone can hang it. I'm starting to think, that something wrong happened between 1.4.0 and 1.5.0 with file-input, but people reporting similar problems use different input plugins... which contradicts my theory. No errors to be seen. According to @ph analysis of my thread dump and what I've seen in --debug spam, file reader threads just vanish over time.

by reader threads I mean those that report this, while debug is enabled:

_discover_file_glob: /PROD/logs/AdminServer/AdminServer.log: glob is: ["/PROD/logs/AdminServer/AdminServer.log"] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}
/TEST/logs/soa_server2/soa_server2.log: file grew, old size 141856, new size 142116 {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"96", :method=>"each"}
Received line {:path=>"/TEST/logs/soa_server2/soa_server2.log", :text=>"####<May 20, 2015 8:07:45 AM CEST> <Info> <Health> <GKKT-SOA-TEST2> <soa_server2> <weblogic.GCMonitor> <<anonymous>> <> <7e0c6a05ce1c3e7a:72530d1:14d6fa861fa:-8000-0000000000005968> <1432102065443> <BEA-310002> <67% of the total memory in the server is free> ", :level=>:debug, :file=>"logstash/inputs/file.rb", :line=>"137", :method=>"run"}

_discover_file_glob: /PROD/nodemgrs/GKKT-SOA-PROD1/logs/nodemanager.log: glob is: [] {:level=>:debug, :file=>"filewatch/watch.rb", :line=>"132",
jsvd commented 9 years ago

@Centopus I'm trying to replicate this as well. My test:

environment: macosx, java 8, logstash 1.5, elasticsearch 1.4.4 data: 9 files of 300k apache log events

logstash config (gist): file -> grok -> date -> geoip -> useragent -> elasticsearch (node)

results:

all 9x300k (2.7 million) documents were inserted and took about 50 minutes:

% curl "http://127.0.0.1:9200/logstash-*/_count"
{"count":2700000,"_shards":{"total":155,"successful":155,"failed":0}}

marvel: screen shot 2015-05-28 at 14 49 33

So..no problems in this scenario.

My next test will be similar but instead I'll send 4x300k from files and 4x300k from tcp socket.

ph commented 9 years ago

@Centopus I think we are missing something in our test, we did not test with a NFS volume, we will do a test and check if it may impact the file input.

Centopus commented 9 years ago

Tomorrow I'm going to try to load logstash with a big bulky amount of logs, to see if that can crash it right away. Up to now I'm just connecting it to production servers and feeding it with the 'normal' data.

That 1.7 java test was still alive when I was leaving work.

And for my OS, I'm running this on OracleLinux (Oracle Redhat fork) on OVM virtual machines inside an Exalogic box, so every filesystem I have available is a NFS (a fast one, but NFS).

Also my production ES database (on 2 cluster nodes) takes up 70GB(60 days) soo its not too large. I'll lookup the daily document counts tomorrow.

My daily document intake is 400k / 300MB, so not this much. And the hang ups do not happen on high load, still have no idea what causes them: http://pasteboard.co/OC7WeDO.png Yesterday (the 1.7 java test) it hung in the evening... when the traffic was already falling down.

I'll try loading it with some bulky data today, to see, if it can cause immediate hang.

I've directed it to archival logs... after 45 minutes of hard work it digested 3M of documents (kopf reports, that the index is at 2GB now). Its still working itself through it... no signs of hanging. Therefore I'm suggesting, that load is not the cause of the problem. It must be something that occurs over time...

There is one thing, that does not happen while reading archival logs, that does happen on live logs.... ROTATION! Going to check this on Monday. Need to think of a way to prepare a reasonable test....

If it were that... it should lose the files one by one, while the're rotating... and not everything.... (and tcp input should be unaffected) Going to check it anyway.

arabold commented 9 years ago

I'm running into a similar problem with Logstash 1.5.0. I'm running it inside of a Docker container so it's a bit more tricky, but after a while (typically a few hours) it simply stops processing logs. No log output, nothing indicating a problem - it just stops and hangs there. Restarting will work. I have not seen this issue with Logstash 1.4.2 before though I was using a much simpler config - as a next step I try to downgrade and see what happens.

My Logstash setup however is not writing to Elasticsearch directly as yours, but only to an Amazon SQS queue. Another instance is then pulling it out of the queue and writes it into Elasticsearch. This second instance didn't stop so far - but it also doesn't do anything other than reading the queue and writing to Elasticsearch. No filters are applied whatsoever. The first instance, the one feeding into SQS, that is the one showing problems.

My instance has 3 inputs:

Then there are a couple of grok and mutate filters. And finally it writes into SQS.

After Logstash dies (or freezes might be the better word), the input plugins seem to not do anything anymore. The sincedb file remains unchanged.

Here's my config as a reference:

input {
  tcp {
    type => "tcp"
    port => 5000
  }

  syslog {
    type => "syslog"
    port => 5544
    tags => []
  }

  s3 {
    type => "elb"
    # FIXME Credential file doesn't work (https://github.com/logstash-plugins/logstash-input-s3/issues/28)
    #aws_credentials_file => "/etc/logstash/conf.d/aws_credentials.yml"
    credentials => [ "...", "..." ]
    bucket => "..."
    prefix => ".../"
    region => "us-east-1"
    sincedb_path => "/usr/share/logstash/data/sincedb/s3-elb.sincedb"
    tags => [ "elb", "aws", "prod" ]
  }

  s3 {
    type => "cloudfront"
    #aws_credentials_file => "/etc/logstash/conf.d/aws_credentials.yml"
    credentials => [ "...", "..." ]
    bucket => "..."
    prefix => ".../"
    region => "us-east-1"
    sincedb_path => "/usr/share/logstash/data/sincedb/s3-cf.sincedb"
    tags => [ "cloudfront", "aws", "prod" ]
  }

}

# Add your filters here
filter {

  # Process logs from AWS Elastic Loadbalancers
  if [type] == "elb" {
    grok {
      match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:loadbalancer} %{IP:clientip}:%{INT:clientport:int} (%{IP:host}:%{INT:hostport:int}|-) %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} %{NUMBER:elb_status_code:int} %{NUMBER:backend_status_code:int} %{NUMBER:received_bytes:int} %{NUMBER:sent_bytes:int} \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\"" ]
    }
    date {
      match => [ "timestamp", "ISO8601" ]
    }
  }

  # Process logs from AWS Cloudfront
  if [type] == "cloudfront" {
    csv {
      separator => "    "
      columns => [ "date", "time", "x-edge-location", "sc-bytes", "clientip", "verb", "cs-host", "request", "response", "referrer", "agent", "cs-uri-query", "cookie", "x-edge-result-type", "x-edge-request-id", "host", "protocol", "cs-bytes", "time-taken" ]
      add_field => [ "listener_timestamp", "%{date} %{time}" ]
    }
    mutate {
      convert => { "response" => "integer" }
    }
    date {
      match => [ "listener_timestamp", "yy-MM-dd HH:mm:ss" ]
      remove_field => [ "listener_timestamp" ]
    }

    urldecode {
      field => "agent"
    }

    if ! [host] {
      mutate {
        update => { "host" => "%{cs-host}" }
      }
    }
  }

  if [type] == "syslog" {

    # Parse syslog date/time patterns
    date {
      match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
      remove_field => [ "timestamp" ]
    }

    # We split the program string received into individual tags. Parts must be separated by a period (".").
    mutate {
      split => [ "program", "." ]
    }
    ruby {
      # FIXME One would think there should be an easier way to merge two arrays with error handling
      code => "event['tags'] = event['program'] + event['tags'] rescue event['program']"
    }

    if "apache" in [program] {
      grok {
        match => [
          # Workaround: For localhost requests the %{COMBINEDAPACHELOG} fails. Thus we handle this here ourselfes.
          "message", "(%{IPORHOST:clientip}|-) %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response:int} (?:%{NUMBER:bytes:int}|-) %{QS:referrer} %{QS:agent}",
          "message", "\[(?<timestamp>.*?)\] (\[(%{WORD:errtype}:)?%{LOGLEVEL:loglevel}\] )?(\[pid %{INT:pid:int}\] )?(\[client %{IPORHOST:clientip}(:%{INT:clientport:int})?\] )?%{GREEDYDATA:errormsg}"
        ]
      }

      date {
        # Try to pull the timestamp from the 'timestamp' field (parsed above with
        # grok). The apache time format looks like: "18/Aug/2011:05:44:34 -0700",
        # the error time format like: "Wed Jun 26 22:13:22 2013"
        match => [
          "timestamp", "dd/MMM/yyyy:HH:mm:ss Z", 
          "EEE MMM dd HH:mm:ss.SSSSSS YYYY", 
          "EEE MMM dd HH:mm:ss.SSS YYYY" , 
          "EEE MMM dd HH:mm:ss YYYY" ]
        remove_field => [ "timestamp" ]
      }
    }

    if "pm2" in [program] {
      mutate {
        # Get rid of color codes
        gsub => [ "message", "\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]", "" ]
      }
    }

    # Parse JSON logs
    if "json" in [program] {
      json {
        source => "message"
      }

      date {
        match => [ "timestamp", "ISO8601" ]
        remove_field => [ "timestamp" ]
      }
    }

    # Parse textual logs (multiline)
    # This parser is kept very generic and should work with PHP logs, Node.js logs as well as anything
    # else you throw at it, as long as it follows somewhat a common pattern:
    # <TIMESTAMP> | <PARAM1> | <PARAM2> | ... | <MESSAGE>
    # <TIMESTAMP>: <PARAM1> | <PARAM2> | ... | <MESSAGE>
    # [...][...] <PARAM1> | <PARAM2> | ... | <MESSAGE>
    if "text" in [program] {

      # Concat multiline logs by checking for a timestamp 
      # (either "[" or something like YYYY-dd-mm) at the beginning of the string
      multiline {
        pattern => "^(\[|\d{4}-\d{2}-\d{2})"
        what => "previous"
        negate => true
      }

      # Messages are split into three parts: <log_timestamp>, <message_fields> and <message_body>
      # This simplifies subsequent parsing.
      grok {
        break_on_match => true
        match => [
          "message", "^\[(?<log_timestamp>.*?)\](\s*\[.*?\])*\s*(?<message_fields>(.*\|)*)\s?(?<message_body>(.|\n|\r)*)",
          "message", "^(?<log_timestamp>\d{4}-\d{2}-\d{2}[-\sT]\d{2}:\d{2}:\d{2})([:\.]\d{2,3})?[:\-\sZ](\s*\[.*?\])*\s*(?<message_fields>(.*\|)*)\s?(?<message_body>(.|\n|\r)*)"
        ]
        add_tag => [ "parse_newline" ]
        tag_on_failure => []
      }

      if "parse_newline" in [tags] {
        # Parse the timestamp (<log_timestamp>)
        date {
          match => [
            "log_timestamp", "EEE, dd MMM YY HH:mm:ss Z", 
            "yyyy-MM-dd-HH:mm:ss", "yyyy-MM-dd HH:mm:ss", 
            "yyyy/MM/dd HH:mm:ss zzz", "yyyy/MM/dd HH:mm:ss", 
            "ISO8601" ]
          remove_field => [ "log_timestamp" ]
        }

        # Parse the message fields (<message_fields>)
        grok {
          break_on_match => false
          match => [
            "message_fields", "(\b)uid: %{INT:uid}",
            "message_fields", "(\b)session_id: %{WORD:session_id}",
            "message_fields", "(\b)uri: %{URIPATH:request}",
            "message_fields", "(\b)url: %{URIPATH:request}",
            "message_fields", "(\b)method: %{WORD:verb}",
            "message_fields", "(\b)user_agent: (?<agent>[^\|]+)\s"
          ]
          remove_field => [ "message_fields" ]
          tag_on_failure => []
        }

        if ![log_timestamp] and ![message_fields] and [message_body] {
          # Shorten message string to it's body only
          mutate {
            replace => [ "message", "%{message_body}" ]
            remove_field => [ "message_body" ]
          }
        }

        mutate {
          remove_tag => [ "parse_newline" ]
        }
      }

      # Process JSON-formatted message strings (this isn't very accurate as all we're doing
      # is to check for strings starting with "{" and ending with "}", but it should be good enough)
      grok {
        match => {
          "message" => "(?m)^\s*(?<message_json>{.*?})\s*$"
        }
        tag_on_failure => []
      }
      json {
        # If we identified the title as JSON string, parse it and remove the message field
        source => "message_json"
        remove_field => [ "message", "message_json" ]
      }

    }
  }

  # Some generic post-processing filters that apply to all types
  #--------------------------------------------------------------

  # Process GeoIP information
  if [clientip] {
    geoip {
      source => "clientip"
      add_tag => [ "geoip" ]
    }
  }

  # Parse HTTP User-Agent strings
  if [agent] {
    useragent {
      source => "agent"
    }
  }

  # Parse HTTP request query parameters
  if [request] {
    grok {
      match => {
        "request" => "^(http(s)?://(?<request_host>[^/\?\#:]*)(:%{NUMBER:request_hostport:int})?)?(?<request_path>[^\?\#]*)(\?(?<request_query>[^#]*))?(#(?<request_fragment>.*)?)?"
      }
      tag_on_failure => []
    }
    kv {
      source => "request_query"
      target => "request_query"
      field_split => "&"
      value_split => "="
    }
  }
}

output {

  sqs {
    aws_credentials_file => "/etc/logstash/conf.d/aws_credentials.yml"
    queue => "Logstash"
    region => "us-east-1"
  }

  # Trying to debug things? Enable the stdout output plugin
  #stdout {
  #  codec => rubydebug
  #}
}
arabold commented 9 years ago

Just as a small update: 1.4.2 is running since a couple of hours now and seems stable. I've already pushed 1.7mio docs and 2.81GB data through it. Next week I'll try to figure out (by testing) if this can be blamed on the network (syslog) input or some of the filters.

parabolic commented 9 years ago

Just a quick update , we've migrated to rabbitmq and 1.5.0 doesn't hang when using the rabbitmq input. Before we where using the lumberjack input.

ianmacl commented 9 years ago

I've been seeing the same issue. I have a new node running Logstash 1.5 out of necessity because 1.4.2 didn't properly support the split filter. I have a udp input, two statsd outputs and one elasticsearch output. It will typically run for a couple of days and then freeze/hang. You can see in the graph below the record counts and where it hangs until I go in and restart it. I just removed the statsd outputs from my logstash config and if it happens again I will paste the config here.

image

vankhoa011 commented 9 years ago

I also got same problem . The logstash is hang after run some days. When the logstash forwarder send a big logs on logstash suddenly, So the logstash seem can't handle these logs and it is die.

My logstash version is 1.5 too.

jordansissel commented 9 years ago

For folks who are observing this hang, can you link to the last 100-200 lines of your Logstash log? Also attach your config if you haven't already. Thanks! :)

arabold commented 9 years ago

In my case, as with the original report of this issue, the logs are empty and give no indication. With empty I literally meanempty.

ianmacl commented 9 years ago

Mine was empty also. I've turned logging up to verbose (debug was creating too much data but maybe I should byte the bullet and do it). I made some changes to my config also and I'm now waiting for it to stop again. I'll post here when it does.

ph commented 9 years ago

People new to this issue, can you also add a gist of your configuration?

surlypants commented 9 years ago

coming in from: https://discuss.elastic.co/t/logstash-silently-stops-processing-events-and-does-not-respond-to-sigterm/1720/3

gist: https://gist.github.com/surlypants/2e1ef99cbc4e2136aff1

surlypants commented 9 years ago

ps ax |grep logstash |grep java

2293 ? SNl 344:17 java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xmx2g -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log -w 6

jstack -l 2293

2293: Unable to open socket file: target process not responding or HotSpot VM not loaded The -F option can be used when the target process is not responding

jstack -F -l 2293 > /tmp/jstack.2293.txt 2>&1

Attaching to process ID 2293, please wait... Debugger attached successfully. Server compiler detected. JVM version is 24.79-b02 Deadlock Detection:

java.lang.RuntimeException: Unable to deduce type of thread from address 0x0000000001923800 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread) at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:162) at sun.jvm.hotspot.runtime.Threads.first(Threads.java:150) at sun.jvm.hotspot.runtime.DeadlockDetector.createThreadTable(DeadlockDetector.java:149) at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:56) at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.tools.jstack.JStack.runJStackTool(JStack.java:136) at sun.tools.jstack.JStack.main(JStack.java:102) Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x0000000001923800 at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62) at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80) at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:158) ... 15 more Can't print deadlocks:Unable to deduce type of thread from address 0x0000000001923800 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread) Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at sun.tools.jstack.JStack.runJStackTool(JStack.java:136) at sun.tools.jstack.JStack.main(JStack.java:102) Caused by: sun.jvm.hotspot.utilities.AssertionFailure: Expecting GenCollectedHeap, G1CollectedHeap, or ParallelScavengeHeap, but got sun.jvm.hotspot.gc_interface.CollectedHeap at sun.jvm.hotspot.utilities.Assert.that(Assert.java:32) at sun.jvm.hotspot.oops.ObjectHeap.collectLiveRegions(ObjectHeap.java:604) at sun.jvm.hotspot.oops.ObjectHeap.iterateSubtypes(ObjectHeap.java:417) at sun.jvm.hotspot.oops.ObjectHeap.iterateObjectsOfKlass(ObjectHeap.java:260) at sun.jvm.hotspot.runtime.ConcurrentLocksPrinter.fillLocks(ConcurrentLocksPrinter.java:70) at sun.jvm.hotspot.runtime.ConcurrentLocksPrinter.(ConcurrentLocksPrinter.java:36) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:61) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) ... 6 more

ph commented 9 years ago

@surlypants Can you gist the full stacktrace ?

surlypants commented 9 years ago

Sure (when it croaks again). You want an strace -f or something else?

ph commented 9 years ago

@surlypants More the better, post them to a gist and include them here.

surlypants commented 9 years ago

here was an strace -f that I let run for a few seconds during that last occurrence...

https://gist.github.com/surlypants/2c4d6087d2d65425dc5c

(and it's since been forcibly restarted -- and croaked again)