search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
14.21k stars 3.5k forks source link

1.5.2 fails to shutdown #3575

Closed tootedom closed 8 years ago

tootedom commented 9 years ago

Hi there,

We've hit the following issue with logstash failing to shutdown on our lumberjack indexing node. Our architecture is:

logstash (shipper) file inputs to a lumberjack output ---------> logstash (indexer) lumberjack input port 555, filters, output ----> elasticsearch

When restarting the logstash (indexer) instance that is listening on the lumberjack input

service logstash restart

The result is that the pipeline shuts down, but the process isn't ended (see the SIGTERM in the below). This is not occurring all the time, and we've only hit this issue twice:

{:timestamp=>"2015-07-08T05:51:33.214000+0000", :message=>"Lumberjack input: the pipeline is blocked, temporary refusing new connection.", :level=>:warn}
{:timestamp=>"2015-07-08T05:51:33.779000+0000", :message=>"CircuitBreaker::Close", :name=>"Lumberjack input", :level=>:warn}
{:timestamp=>"2015-07-08T07:39:56.617000+0000", :message=>"SIGTERM received. Shutting down the pipeline.", :level=>:warn}
{:timestamp=>"2015-07-08T07:39:58.635000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2015-07-08T07:39:58.635000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2015-07-08T07:39:58.642000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2015-07-08T07:39:58.717000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2015-07-08T07:39:58.718000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2015-07-08T07:39:58.718000+0000", :message=>"CircuitBreaker::Open", :name=>"Lumberjack input", :level=>:warn}

The result is that you have client still connected to SIGTERM'd pipeline, as the process is still up, but it is doing nothing. It is not accepting data or sending any data to any output (elasticsearch in out case):

tcp      153      0 ::ffff:xxx.xxx.xxx.xxx:5555     ::ffff:xxx.xxx.xxx.xxx:39454    ESTABLISHED -
tcp       83      0 ::ffff:xxx.xxx.xxx.xxx:5555     ::ffff:xxx.xxx.xxx.xxx:35446  ESTABLISHED -
tcp        0      0 ::ffff:xxx.xxx.xxx.xxx:5555     ::ffff:xxx.xxx.xxx.xxx:35274 ESTABLISHED 10881/java
tcp        0      0 ::ffff:xxx.xxx.xxx.xxx:5555     ::ffff:xxx.xxx.xxx.xxx:56818 ESTABLISHED 10881/java
tcp        0      0 ::ffff:xxx.xxx.xxx.xxx:5555     ::ffff:xxx.xxx.xxx.xxx:7731  ESTABLISHED 10881/java
tcp        0      0 ::ffff:xxx.xxx.xxx.xxx:5555     ::ffff:xxx.xxx.xxx.xxx:28076  ESTABLISHED 10881/java
tcp        0      0 ::ffff:xxx.xxx.xxx.xxx:5555     ::ffff:xxx.xxx.xxx.xxx:37296  ESTABLISHED 10881/java

The additional issue is the the restart script (sysv init script), does not notice that the process isn't killed, and attempts to start another logstash instance, which you can see from the log (failure to bind to 5555 as the old process is still up and is bound to that port):

{:timestamp=>"2015-07-08T10:16:34.669000+0000", :message=>"The error reported is: \n  Address already in use - bind - Address already in use"}

Apologies I do not have a thread dump or heap for you. I shall attempt to get this the next time we hit this issue (This is the 2nd time we've seen this).

djsly commented 9 years ago

+1

We hare the same issue, Our Logstash instance is taking some load form 40-50 machines. We run the official Docker images and when we do docker-compose kill, the container kill process hangs. We need to restart the docker service, which then creates a dead container.

To top it off with the latest release of docker-compose, dead containers break the usage of docker-compose (fix in process). So we have no choice but to restart the host to get ride of the defunct process...

Please, try fixing this issue as it is becoming more then annoying.

colinsurprenant commented 9 years ago

Thanks for the report guys.

@tootedom looking at your log traces, since the lumberjack input hits the circuit breaker this indicates that the downstream output plugins are getting stalled somehow which in turn prevents the proper propagation of the shutdown event. It seems you are in fact facing the general pipeline shutdown stall condition described in #3451

Note that we are working on a fix for this problem for 1.6. We will also create a meta-issue about that to link the related issues to the shutdown stalling bigger problem and track progress.

nightwolfzor commented 9 years ago

+1 seeing this issue as well. Logs are also being spammed with 

{:timestamp=>"2015-07-13T01:14:15.154000+0000", :message=>"CircuitBreaker::rescuing exceptions", :name=>"Lumberjack input", :exception=>LogStash::SizedQueueTimeout::TimeoutError, :level=>:warn}
{:timestamp=>"2015-07-13T01:14:15.155000+0000", :message=>"CircuitBreaker::Open", :name=>"Lumberjack input", :level=>:warn}
{:timestamp=>"2015-07-13T01:14:15.157000+0000", :message=>"Exception in lumberjack input thread", :exception=>#<LogStash::CircuitBreaker::OpenBreaker: for Lumberjack input>, :level=>:error}
{:timestamp=>"2015-07-13T01:14:15.454000+0000", :message=>"CircuitBreaker::Open", :name=>"Lumberjack input", :level=>:warn}
{:timestamp=>"2015-07-13T01:14:15.455000+0000", :message=>"Exception in lumberjack input thread", :exception=>#<LogStash::CircuitBreaker::OpenBreaker: for Lumberjack input>, :level=>:error}
{:timestamp=>"2015-07-13T01:15:02.262000+0000", :message=>"CircuitBreaker::Close", :name=>"Lumberjack input", :level=>:warn}
tootedom commented 9 years ago

Hi there, we just hit this on another environment (where logstash fails to shutdown, and a second process is started), here is a stack trace.

Interesting information on this node is that the output (lumberjack - The logstash instance is shipping logs to a Lumberjack input on another node), is that the connection to other node (the lumberjack input) is stuck in SYN_SENT (the firewall is dropping the traffic - something we need to fix our end); but perhaps this an easy way to replicate unable to shutdown (will try when I have a moment).

tcp        0      1 ::ffff:xx.xx.xx.xx:56053     ::ffff:xx.xx.xx.xx:5555  SYN_SENT    2102/java
    $ sudo /usr/java/default/bin/jstack -F 2102
    [sudo] password for xxxxx: 
    Attaching to process ID 2102, please wait...
    Debugger attached successfully.
    Server compiler detected.
    JVM version is 25.45-b02
    Deadlock Detection:

    No deadlocks found.

    Thread 2772: (state = BLOCKED)
     - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
     - java.lang.Object.wait(long, int) @bci=58, line=460 (Interpreted frame)
     - org.jruby.RubyThread$SleepTask.run() @bci=19, line=1049 (Interpreted frame)
     - org.jruby.RubyThread.executeBlockingTask(org.jruby.RubyThread$BlockingTask) @bci=14, line=1065 (Interpreted frame)
     - org.jruby.RubyThread.wait_timeout(org.jruby.runtime.builtin.IRubyObject, java.lang.Double) @bci=88, line=1413 (Interpreted frame)
     - org.jruby.ext.thread.SizedQueue.push(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject) @bci=47, line=133 (Interpreted frame)
     - org.jruby.ext.thread.SizedQueue$INVOKER$i$1$0$push.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.builtin.IRubyObject) @bci=7 (Interpreted frame)
     - org.jruby.runtime.callsite.CachingCallSite.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject) @bci=38, line=168 (Compiled frame)
     - org.jruby.ast.CallOneArgNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=30, line=57 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(org.jruby.Ruby, org.jruby.runtime.ThreadContext, java.lang.String, int, org.jruby.ast.Node, java.lang.String, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=30, line=112 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.evalBlockBody(org.jruby.runtime.ThreadContext, org.jruby.runtime.Binding, org.jruby.runtime.builtin.IRubyObject) @bci=25, line=206 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.yield(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, boolean, org.jruby.runtime.Binding, org.jruby.runtime.Block$Type, org.jruby.runtime.Block) @bci=51, line=194 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Binding, org.jruby.runtime.Block$Type, org.jruby.runtime.Block) @bci=16, line=125 (Compiled frame)
     - org.jruby.runtime.Block.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Block) @bci=15, line=101 (Compiled frame)
     - org.jruby.RubyProc.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=68, line=290 (Compiled frame)
     - org.jruby.RubyProc.call19(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Block) @bci=48, line=271 (Compiled frame)
     - org.jruby.RubyProc$INVOKER$i$0$0$call19.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Block) @bci=9 (Compiled frame)
     - org.jruby.internal.runtime.methods.DynamicMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.Block) @bci=11, line=202 (Compiled frame)
     - org.jruby.internal.runtime.methods.DynamicMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String) @bci=9, line=198 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject) @bci=36, line=134 (Compiled frame)
     - org.jruby.ast.CallNoArgNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=18, line=60 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.ast.BlockNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=33, line=71 (Compiled frame)
     - org.jruby.ast.IfNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=98, line=118 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.ast.BlockNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=33, line=71 (Compiled frame)
     - org.jruby.ast.WhileNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=46, line=131 (Interpreted frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.ast.BlockNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=33, line=71 (Compiled frame)
     - org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(org.jruby.Ruby, org.jruby.runtime.ThreadContext, java.lang.String, int, org.jruby.RubyModule, org.jruby.ast.Node, java.lang.String, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block, boolean) @bci=30, line=74 (Compiled frame)
     - org.jruby.internal.runtime.methods.InterpretedMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=79, line=204 (Compiled frame)
     - org.jruby.internal.runtime.methods.DefaultMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=24, line=211 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyClass, org.jruby.runtime.Block, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject) @bci=61, line=336 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.callBlock(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=54, line=179 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.callIter(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=8, line=188 (Compiled frame)
     - org.jruby.ast.CallOneArgBlockNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=43, line=60 (Interpreted frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(org.jruby.Ruby, org.jruby.runtime.ThreadContext, java.lang.String, int, org.jruby.ast.Node, java.lang.String, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=30, line=112 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.evalBlockBody(org.jruby.runtime.ThreadContext, org.jruby.runtime.Binding, org.jruby.runtime.builtin.IRubyObject) @bci=25, line=206 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.yield(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, boolean, org.jruby.runtime.Binding, org.jruby.runtime.Block$Type, org.jruby.runtime.Block) @bci=51, line=194 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Binding, org.jruby.runtime.Block$Type, org.jruby.runtime.Block) @bci=16, line=125 (Compiled frame)
     - org.jruby.runtime.Block.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Block) @bci=15, line=101 (Compiled frame)
     - org.jruby.RubyProc.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=68, line=290 (Compiled frame)
     - org.jruby.RubyProc.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[]) @bci=7, line=228 (Interpreted frame)
     - org.jruby.internal.runtime.RubyRunnable.run() @bci=146, line=99 (Interpreted frame)
     - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

    Thread 2771: (state = BLOCKED)
     - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
     - java.lang.Object.wait(long, int) @bci=58, line=460 (Interpreted frame)
     - org.jruby.RubyThread$SleepTask.run() @bci=19, line=1049 (Interpreted frame)
     - org.jruby.RubyThread.executeBlockingTask(org.jruby.RubyThread$BlockingTask) @bci=14, line=1065 (Interpreted frame)
     - org.jruby.RubyThread.wait_timeout(org.jruby.runtime.builtin.IRubyObject, java.lang.Double) @bci=88, line=1413 (Interpreted frame)
     - org.jruby.ext.thread.SizedQueue.push(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject) @bci=47, line=133 (Interpreted frame)
     - org.jruby.ext.thread.SizedQueue$INVOKER$i$1$0$push.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.builtin.IRubyObject) @bci=7 (Interpreted frame)
     - org.jruby.runtime.callsite.CachingCallSite.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject) @bci=38, line=168 (Compiled frame)
     - org.jruby.ast.CallOneArgNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=30, line=57 (Compiled frame)
     - org.jruby.ast.IfNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=98, line=118 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(org.jruby.Ruby, org.jruby.runtime.ThreadContext, java.lang.String, int, org.jruby.ast.Node, java.lang.String, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=30, line=112 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.evalBlockBody(org.jruby.runtime.ThreadContext, org.jruby.runtime.Binding, org.jruby.runtime.builtin.IRubyObject) @bci=25, line=206 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.yield(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Binding, org.jruby.runtime.Block$Type) @bci=43, line=157 (Compiled frame)
     - org.jruby.runtime.Block.yield(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject) @bci=14, line=142 (Compiled frame)
     - org.jruby.RubyArray.eachCommon(org.jruby.runtime.ThreadContext, org.jruby.runtime.Block) @bci=41, line=1606 (Compiled frame)
     - org.jruby.RubyArray.each(org.jruby.runtime.ThreadContext, org.jruby.runtime.Block) @bci=10, line=1613 (Compiled frame)
     - org.jruby.RubyArray$INVOKER$i$0$0$each.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.Block) @bci=7 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.callBlock(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=38, line=143 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.callIter(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=6, line=154 (Compiled frame)
     - org.jruby.ast.CallNoArgBlockNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=35, line=64 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.ast.WhenOneArgNode.whenSlowTest(org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.ThreadContext, org.jruby.Ruby, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=46, line=36 (Compiled frame)
     - org.jruby.ast.WhenOneArgNode.when(org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.ThreadContext, org.jruby.Ruby, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=33, line=46 (Compiled frame)
     - org.jruby.ast.CaseNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=111, line=133 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.ast.BlockNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=33, line=71 (Compiled frame)
     - org.jruby.ast.WhileNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=46, line=131 (Interpreted frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.ast.RescueNode.executeBody(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=21, line=221 (Compiled frame)
     - org.jruby.ast.RescueNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=26, line=116 (Compiled frame)
     - org.jruby.ast.BeginNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=9, line=83 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.ast.BlockNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=33, line=71 (Compiled frame)
     - org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(org.jruby.Ruby, org.jruby.runtime.ThreadContext, java.lang.String, int, org.jruby.RubyModule, org.jruby.ast.Node, java.lang.String, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block, boolean) @bci=30, line=74 (Compiled frame)
     - org.jruby.internal.runtime.methods.InterpretedMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String) @bci=80, line=139 (Compiled frame)
     - org.jruby.internal.runtime.methods.DefaultMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String) @bci=20, line=187 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyClass, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject) @bci=53, line=306 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject) @bci=46, line=136 (Compiled frame)
     - org.jruby.ast.VCallNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=7, line=88 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(org.jruby.Ruby, org.jruby.runtime.ThreadContext, java.lang.String, int, org.jruby.ast.Node, java.lang.String, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=30, line=112 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.evalBlockBody(org.jruby.runtime.ThreadContext, org.jruby.runtime.Binding, org.jruby.runtime.builtin.IRubyObject) @bci=25, line=206 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.yield(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, boolean, org.jruby.runtime.Binding, org.jruby.runtime.Block$Type, org.jruby.runtime.Block) @bci=51, line=194 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Binding, org.jruby.runtime.Block$Type, org.jruby.runtime.Block) @bci=16, line=125 (Compiled frame)
     - org.jruby.runtime.Block.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Block) @bci=15, line=101 (Compiled frame)
     - org.jruby.RubyProc.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=68, line=290 (Compiled frame)
     - org.jruby.RubyProc.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[]) @bci=7, line=228 (Interpreted frame)
     - org.jruby.internal.runtime.RubyRunnable.run() @bci=146, line=99 (Interpreted frame)
     - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

    Thread 2676: (state = BLOCKED)
     - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
     - org.jruby.RubyThread.sleep(long) @bci=66, line=1001 (Compiled frame)
     - org.jruby.RubyKernel.sleep(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject[]) @bci=131, line=835 (Compiled frame)
     - org.jruby.RubyKernel$INVOKER$s$0$1$sleep.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.builtin.IRubyObject[]) @bci=29 (Compiled frame)
     - org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=15, line=669 (Compiled frame)
     - org.jruby.internal.runtime.methods.DynamicMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.builtin.IRubyObject) @bci=11, line=206 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject) @bci=38, line=168 (Compiled frame)
     - org.jruby.ast.FCallOneArgNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=19, line=36 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.ast.BlockNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=33, line=71 (Compiled frame)
     - org.jruby.ast.BlockNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=33, line=71 (Compiled frame)
     - org.jruby.ast.RescueBodyNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=9, line=108 (Compiled frame)
     - org.jruby.ast.RescueNode.handleException(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block, org.jruby.exceptions.RaiseException) @bci=66, line=174 (Compiled frame)
     - org.jruby.ast.RescueNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=61, line=120 (Compiled frame)
     - org.jruby.ast.BeginNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=9, line=83 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.ast.BlockNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=33, line=71 (Compiled frame)
     - org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(org.jruby.Ruby, org.jruby.runtime.ThreadContext, java.lang.String, int, org.jruby.RubyModule, org.jruby.ast.Node, java.lang.String, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block, boolean) @bci=30, line=74 (Compiled frame)
     - org.jruby.internal.runtime.methods.InterpretedMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String) @bci=80, line=139 (Compiled frame)
     - org.jruby.internal.runtime.methods.DefaultMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String) @bci=20, line=187 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyClass, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject) @bci=53, line=306 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject) @bci=46, line=136 (Compiled frame)
     - org.jruby.ast.VCallNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=7, line=88 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.ast.BlockNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=33, line=71 (Compiled frame)
     - org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(org.jruby.Ruby, org.jruby.runtime.ThreadContext, java.lang.String, int, org.jruby.RubyModule, org.jruby.ast.Node, java.lang.String, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block, boolean) @bci=30, line=74 (Compiled frame)
     - org.jruby.internal.runtime.methods.InterpretedMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Block) @bci=100, line=112 (Compiled frame)
     - org.jruby.internal.runtime.methods.DefaultMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Block) @bci=24, line=169 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyClass, org.jruby.runtime.Block, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject) @bci=61, line=286 (Interpreted frame)
     - org.jruby.runtime.callsite.CachingCallSite.callBlock(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Block) @bci=54, line=81 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Block) @bci=8, line=85 (Compiled frame)
     - org.jruby.RubySymbol$1.yieldInner(org.jruby.runtime.ThreadContext, org.jruby.RubyArray, org.jruby.runtime.Block) @bci=38, line=461 (Compiled frame)
     - org.jruby.RubySymbol$1.yield(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Binding, org.jruby.runtime.Block$Type) @bci=14, line=481 (Compiled frame)
     - org.jruby.runtime.Block.yield(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject) @bci=14, line=142 (Compiled frame)
     - org.jruby.RubyArray.eachCommon(org.jruby.runtime.ThreadContext, org.jruby.runtime.Block) @bci=41, line=1606 (Compiled frame)
     - org.jruby.RubyArray.each(org.jruby.runtime.ThreadContext, org.jruby.runtime.Block) @bci=10, line=1613 (Compiled frame)
     - org.jruby.RubyArray$INVOKER$i$0$0$each.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.Block) @bci=7 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyClass, org.jruby.runtime.Block, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject) @bci=57, line=316 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.callBlock(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=50, line=145 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=6, line=149 (Compiled frame)
     - org.jruby.ast.CallNoArgBlockPassNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=30, line=53 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.ast.BlockNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=33, line=71 (Compiled frame)
     - org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(org.jruby.Ruby, org.jruby.runtime.ThreadContext, java.lang.String, int, org.jruby.RubyModule, org.jruby.ast.Node, java.lang.String, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block, boolean) @bci=30, line=74 (Compiled frame)
     - org.jruby.internal.runtime.methods.InterpretedMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String) @bci=80, line=139 (Compiled frame)
     - org.jruby.internal.runtime.methods.DefaultMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String) @bci=20, line=187 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyClass, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject) @bci=53, line=306 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject) @bci=46, line=136 (Compiled frame)
     - org.jruby.ast.VCallNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=7, line=88 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.ast.BlockNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=33, line=71 (Compiled frame)
     - org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(org.jruby.Ruby, org.jruby.runtime.ThreadContext, java.lang.String, int, org.jruby.RubyModule, org.jruby.ast.Node, java.lang.String, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block, boolean) @bci=30, line=74 (Compiled frame)
     - org.jruby.internal.runtime.methods.InterpretedMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String) @bci=80, line=139 (Compiled frame)
     - org.jruby.internal.runtime.methods.DefaultMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String) @bci=20, line=187 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyClass, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject) @bci=53, line=306 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject) @bci=46, line=136 (Compiled frame)
     - org.jruby.ast.CallNoArgNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=18, line=60 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.ast.BlockNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=33, line=71 (Compiled frame)
     - org.jruby.ast.RescueNode.executeBody(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=21, line=221 (Compiled frame)
     - org.jruby.ast.RescueNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=26, line=116 (Compiled frame)
     - org.jruby.ast.EnsureNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=16, line=96 (Interpreted frame)
     - org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(org.jruby.Ruby, org.jruby.runtime.ThreadContext, java.lang.String, int, org.jruby.RubyModule, org.jruby.ast.Node, java.lang.String, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block, boolean) @bci=30, line=74 (Compiled frame)
     - org.jruby.internal.runtime.methods.InterpretedMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String) @bci=80, line=139 (Compiled frame)
     - org.jruby.internal.runtime.methods.DefaultMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String) @bci=20, line=187 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyClass, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject) @bci=53, line=306 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject) @bci=46, line=136 (Compiled frame)
     - org.jruby.ast.CallNoArgNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=18, line=60 (Compiled frame)
     - org.jruby.ast.ReturnNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=10, line=92 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.ast.BlockNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=33, line=71 (Compiled frame)
     - org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(org.jruby.Ruby, org.jruby.runtime.ThreadContext, java.lang.String, int, org.jruby.ast.Node, java.lang.String, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=30, line=112 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.evalBlockBody(org.jruby.runtime.ThreadContext, org.jruby.runtime.Binding, org.jruby.runtime.builtin.IRubyObject) @bci=25, line=206 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.yield(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, boolean, org.jruby.runtime.Binding, org.jruby.runtime.Block$Type, org.jruby.runtime.Block) @bci=51, line=194 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Binding, org.jruby.runtime.Block$Type, org.jruby.runtime.Block) @bci=16, line=125 (Compiled frame)
     - org.jruby.runtime.Block.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Block) @bci=15, line=101 (Compiled frame)
     - org.jruby.RubyProc.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=68, line=290 (Compiled frame)
     - org.jruby.RubyProc.call19(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Block) @bci=48, line=271 (Compiled frame)
     - org.jruby.RubyProc$INVOKER$i$0$0$call19.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Block) @bci=9 (Compiled frame)
     - org.jruby.internal.runtime.methods.DynamicMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.Block) @bci=11, line=202 (Compiled frame)
     - org.jruby.internal.runtime.methods.DynamicMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String) @bci=9, line=198 (Interpreted frame)
     - org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyClass, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject) @bci=53, line=306 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject) @bci=46, line=136 (Compiled frame)
     - org.jruby.ast.CallNoArgNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=18, line=60 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(org.jruby.Ruby, org.jruby.runtime.ThreadContext, java.lang.String, int, org.jruby.ast.Node, java.lang.String, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=30, line=112 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.evalBlockBody(org.jruby.runtime.ThreadContext, org.jruby.runtime.Binding, org.jruby.runtime.builtin.IRubyObject) @bci=25, line=206 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.yield(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, boolean, org.jruby.runtime.Binding, org.jruby.runtime.Block$Type, org.jruby.runtime.Block) @bci=51, line=194 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Binding, org.jruby.runtime.Block$Type, org.jruby.runtime.Block) @bci=16, line=125 (Compiled frame)
     - org.jruby.runtime.Block.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Block) @bci=15, line=101 (Compiled frame)
     - org.jruby.RubyProc.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=68, line=290 (Compiled frame)
     - org.jruby.RubyProc.call19(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Block) @bci=48, line=271 (Compiled frame)
     - org.jruby.RubyProc$INVOKER$i$0$0$call19.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Block) @bci=9 (Compiled frame)
     - org.jruby.internal.runtime.methods.DynamicMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.Block) @bci=11, line=202 (Compiled frame)
     - org.jruby.internal.runtime.methods.DynamicMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String) @bci=9, line=198 (Interpreted frame)
     - org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyClass, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject) @bci=53, line=306 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject) @bci=46, line=136 (Compiled frame)
     - org.jruby.ast.CallSpecialArgNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=88, line=65 (Interpreted frame)
     - org.jruby.ast.DAsgnNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=17, line=110 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.ast.BlockNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=33, line=71 (Compiled frame)
     - org.jruby.ast.RescueNode.executeBody(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=21, line=221 (Interpreted frame)
     - org.jruby.ast.RescueNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=26, line=116 (Interpreted frame)
     - org.jruby.ast.BeginNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=9, line=83 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(org.jruby.Ruby, org.jruby.runtime.ThreadContext, java.lang.String, int, org.jruby.ast.Node, java.lang.String, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=30, line=112 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.evalBlockBody(org.jruby.runtime.ThreadContext, org.jruby.runtime.Binding, org.jruby.runtime.builtin.IRubyObject) @bci=25, line=206 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.yield(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, boolean, org.jruby.runtime.Binding, org.jruby.runtime.Block$Type, org.jruby.runtime.Block) @bci=51, line=194 (Compiled frame)
     - org.jruby.runtime.Interpreted19Block.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Binding, org.jruby.runtime.Block$Type, org.jruby.runtime.Block) @bci=16, line=125 (Compiled frame)
     - org.jruby.runtime.Block.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Block) @bci=15, line=101 (Compiled frame)
     - org.jruby.RubyProc.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=68, line=290 (Compiled frame)
     - org.jruby.RubyProc.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject[]) @bci=7, line=228 (Interpreted frame)
     - org.jruby.internal.runtime.RubyRunnable.run() @bci=146, line=99 (Interpreted frame)
     - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

    Thread 2675: (state = BLOCKED)
     - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
     - java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=143 (Compiled frame)
     - java.lang.ref.ReferenceQueue.remove() @bci=2, line=164 (Compiled frame)
     - org.jruby.util.ReferenceReaper$1.run() @bci=7, line=32 (Interpreted frame)
     - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

    Thread 2198: (state = BLOCKED)

    Thread 2197: (state = BLOCKED)
     - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
     - java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=143 (Compiled frame)
     - java.lang.ref.ReferenceQueue.remove() @bci=2, line=164 (Compiled frame)
     - java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=209 (Interpreted frame)

    Thread 2196: (state = BLOCKED)
     - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
     - java.lang.Object.wait() @bci=2, line=502 (Interpreted frame)
     - java.lang.ref.Reference$ReferenceHandler.run() @bci=36, line=157 (Interpreted frame)

    Thread 2161: (state = BLOCKED)
     - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
     - java.lang.Thread.join(long) @bci=70, line=1253 (Compiled frame)
     - org.jruby.internal.runtime.NativeThread.join(long) @bci=11, line=75 (Compiled frame)
     - org.jruby.RubyThread.join(org.jruby.runtime.builtin.IRubyObject[]) @bci=201, line=696 (Compiled frame)
     - org.jruby.RubyThread$INVOKER$i$0$1$join.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.builtin.IRubyObject[]) @bci=31 (Interpreted frame)
     - org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.Block) @bci=9, line=665 (Interpreted frame)
     - org.jruby.internal.runtime.methods.DynamicMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String) @bci=9, line=198 (Interpreted frame)
     - org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyClass, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject) @bci=53, line=306 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject) @bci=46, line=136 (Compiled frame)
     - org.jruby.ast.CallNoArgNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=18, line=60 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.ast.BlockNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=33, line=71 (Compiled frame)
     - org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(org.jruby.Ruby, org.jruby.runtime.ThreadContext, java.lang.String, int, org.jruby.RubyModule, org.jruby.ast.Node, java.lang.String, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block, boolean) @bci=30, line=74 (Compiled frame)
     - org.jruby.internal.runtime.methods.InterpretedMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String) @bci=80, line=139 (Compiled frame)
     - org.jruby.internal.runtime.methods.DefaultMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String) @bci=20, line=187 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyClass, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject) @bci=53, line=306 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject) @bci=46, line=136 (Compiled frame)
     - org.jruby.ast.CallNoArgNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=18, line=60 (Compiled frame)
     - org.jruby.ast.FCallOneArgNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=16, line=36 (Compiled frame)
     - org.jruby.ast.NewlineNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=41, line=105 (Compiled frame)
     - org.jruby.ast.BlockNode.interpret(org.jruby.Ruby, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block) @bci=33, line=71 (Compiled frame)
     - org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(org.jruby.Ruby, org.jruby.runtime.ThreadContext, java.lang.String, int, org.jruby.RubyModule, org.jruby.ast.Node, java.lang.String, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.Block, boolean) @bci=30, line=74 (Compiled frame)
     - org.jruby.internal.runtime.methods.InterpretedMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.builtin.IRubyObject) @bci=82, line=182 (Compiled frame)
     - org.jruby.internal.runtime.methods.DefaultMethod.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyModule, java.lang.String, org.jruby.runtime.builtin.IRubyObject) @bci=22, line=203 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(org.jruby.runtime.builtin.IRubyObject, org.jruby.RubyClass, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject) @bci=57, line=326 (Compiled frame)
     - org.jruby.runtime.callsite.CachingCallSite.call(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject) @bci=50, line=170 (Compiled frame)
     - opt.logstash.lib.bootstrap.environment.__file__(opt.logstash.lib.bootstrap.environment, org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, org.jruby.runtime.builtin.IRubyObject[], org.jruby.runtime.Block) @bci=303, line=48 (Interpreted frame)
     - opt.logstash.lib.bootstrap.environment.load(org.jruby.runtime.ThreadContext, org.jruby.runtime.builtin.IRubyObject, boolean) @bci=22 (Interpreted frame)
     - org.jruby.Ruby.runScript(org.jruby.ast.executable.Script, boolean) @bci=12, line=867 (Interpreted frame)
     - org.jruby.Ruby.runScript(org.jruby.ast.executable.Script) @bci=3, line=860 (Interpreted frame)
     - org.jruby.Ruby.runNormally(org.jruby.ast.Node) @bci=73, line=729 (Interpreted frame)
     - org.jruby.Ruby.runFromMain(java.io.InputStream, java.lang.String) @bci=320, line=578 (Interpreted frame)
     - org.jruby.Main.doRunFromMain(org.jruby.Ruby, java.io.InputStream, java.lang.String) @bci=12, line=395 (Interpreted frame)
     - org.jruby.Main.internalRun() @bci=234, line=290 (Interpreted frame)
     - org.jruby.Main.run(java.lang.String[]) @bci=9, line=217 (Interpreted frame)
     - org.jruby.Main.main(java.lang.String[]) @bci=35, line=197 (Interpreted frame)
adrianlop commented 9 years ago

+1 same here

dbrgn commented 9 years ago

Here, also affected.

ikkaro commented 9 years ago

Here, also affected.

vankhoa011 commented 9 years ago

+1

{:timestamp=>"2015-07-21T11:58:38.501000+0000", :message=>"Lumberjack input: the pipeline is blocked, temporary refusing new connection.", :level=>:warn}
{:timestamp=>"2015-07-21T11:58:38.567000+0000", :message=>"CircuitBreaker::Open", :name=>"Lumberjack input", :level=>:warn}
{:timestamp=>"2015-07-21T11:58:38.568000+0000", :message=>"Exception in lumberjack input thread", :exception=>#<LogStash::CircuitBreaker::OpenBreaker: for Lumberjack input>, :level=>:error}
{:timestamp=>"2015-07-21T11:58:38.592000+0000", :message=>"CircuitBreaker::Open", :name=>"Lumberjack input", :level=>:warn}
{:timestamp=>"2015-07-21T11:58:38.592000+0000", :message=>"Exception in lumberjack input thread", :exception=>#<LogStash::CircuitBreaker::OpenBreaker: for Lumberjack input>, :level=>:error}
agrass commented 9 years ago

+1

ghost commented 9 years ago

+1

hzome commented 9 years ago

+1

bluebenno commented 9 years ago

+1 This quite nasty - in my case the (many) upstream systems have a large inrush (logstash-forwarder) of log payload so if you stop logstash it cannot start as the breakers trip. Work around is to run logstash in debug mode (--debug), which seems to throttle enough.

fortysevenlabs commented 9 years ago

+1

pbruna commented 9 years ago

+1

pbruna commented 9 years ago

I can confirm that @bluebenno workaround with --debug works, but the disk was filling quickly so I have to send the logs to /dev/null

bluebenno commented 9 years ago

Using protocol => "transport" works very well and has been extremely stable for me ( no --debug required). YMMV etc. i.e.

    elasticsearch {
        protocol => "transport"
        host => "blah"
        index => "index stuff"
    }
foresightyj commented 9 years ago

+1

agrass commented 9 years ago

the workaround of @bluebenno for the moment works well for me too

pbruna commented 9 years ago

I've upgraded to logstash-1.5.3-1.noarch and everything works again!!

ph commented 9 years ago

@pbruna any reason not to use 1.5.4?

pbruna commented 9 years ago

@ph no really, it just wasn't available at that moment, Im going to give it a try.

linuxmalaysia commented 9 years ago

Hi,

Im using version 1.5.4 ad still having this problem.

{:timestamp=>"2015-09-04T16:25:11.542000+0800", :message=>["INFLIGHT_EVENTS_REPORT", "2015-09-04T16:25:11+08:00", {"input_to_filter"=>20, "filter_to_output"=>0, "outputs"=>[]}], :level=>:warn} {:timestamp=>"2015-09-04T16:25:12.267000+0800", :message=>["INFLIGHT_EVENTS_REPORT", "2015-09-04T16:25:12+08:00", {"input_to_filter"=>20, "filter_to_output"=>0, "outputs"=>[]}], :level=>:warn} {:timestamp=>"2015-09-04T16:25:15.044000+0800", :message=>["INFLIGHT_EVENTS_REPORT", "2015-09-04T16:25:15+08:00", {"input_to_filter"=>20, "filter_to_output"=>0, "outputs"=>[]}], :level=>:warn} {:timestamp=>"2015-09-04T16:25:16.543000+0800", :message=>["INFLIGHT_EVENTS_REPORT", "2015-09-04T16:25:16+08:00", {"input_to_filter"=>20, "filter_to_output"=>0, "outputs"=>[]}], :level=>:warn} {:timestamp=>"2015-09-04T16:25:17.268000+0800", :message=>["INFLIGHT_EVENTS_REPORT", "2015-09-04T16:25:17+08:00", {"input_to_filter"=>20, "filter_to_output"=>0, "outputs"=>[]}], :level=>:warn} {:timestamp=>"2015-09-04T16:25:20.045000+0800", :message=>["INFLIGHT_EVENTS_REPORT", "2015-09-04T16:25:20+08:00", {"input_to_filter"=>20, "filter_to_output"=>0, "outputs"=>[]}], :level=>:warn} {:timestamp=>"2015-09-04T16:25:21.544000+0800", :message=>["INFLIGHT_EVENTS_REPORT", "2015-09-04T16:25:21+08:00", {"input_to_filter"=>20, "filter_to_output"=>0, "outputs"=>[]}], :level=>:warn} {:timestamp=>"2015-09-04T16:25:22.269000+0800", :message=>["INFLIGHT_EVENTS_REPORT", "2015-09-04T16:25:22+08:00", {"input_to_filter"=>20, "filter_to_output"=>0, "outputs"=>[]}], :level=>:warn} {:timestamp=>"2015-09-04T16:25:25.046000+0800", :message=>["INFLIGHT_EVENTS_REPORT", "2015-09-04T16:25:25+08:00", {"input_to_filter"=>20, "filter_to_output"=>0, "outputs"=>[]}], :level=>:warn} {:timestamp=>"2015-09-04T16:25:26.545000+0800", :message=>["INFLIGHT_EVENTS_REPORT", "2015-09-04T16:25:26+08:00", {"input_to_filter"=>20, "filter_to_output"=>0, "outputs"=>[]}], :level=>:warn}

aterribleloss commented 9 years ago

I have also updated to 1.5.4 and am still having the issue as well.

beeva-diegogarcia commented 9 years ago

+1

fizerkhan commented 9 years ago

+1

bzhpwr commented 9 years ago

+1

fizerkhan commented 9 years ago

I fixed it with KILL_ON_STOP_TIMEOUT setting in /etc/default/logstash

# If this is set to 1, then when `stop` is called, if the process has
# not exited within a reasonable time, SIGKILL will be sent next.
# The default behavior is to simply log a message "program stop failed; still running"
KILL_ON_STOP_TIMEOUT=1
# Arguments to pass to logstash agent
LS_OPTS="-w 10"

Also logstash has failed so frequently with following message

{
  :timestamp=>"2015-10-14T00:43:45.970000-0400", 
  :message=>"Lumberjack input: the pipeline is blocked, temporary refusing new connection.", 
  :level=>:warn
}

I fixed this issue temporarily by restarting logstash every 1 hour using cron job

0 * * * * service logstash restart
suyograo commented 8 years ago

2.x added forced shutdown and stall detection. https://www.elastic.co/guide/en/logstash/current/stalled-shutdown.html