logstash-plugins / logstash-output-lumberjack

Apache License 2.0
6 stars 24 forks source link

Lumberjack output fails with java.lang.IllegalArgumentException #5

Open suyograo opened 9 years ago

suyograo commented 9 years ago

from @dsolsona, https://github.com/elastic/logstash/issues/3559

Hi,

I'm seeing this error with the lumberjack output plugin and logstash 1.5.2

{:timestamp=>"2015-07-05T21:56:11.282000-0500", :message=>"Client write error, trying connect", :e=>java.lang.IllegalArgumentException, :backtrace=>["java.nio.Buffer.position(Buffer.java:244)", "java.nio.DirectByteBuffer.put(DirectByteBuffer.java:343)", "sun.nio.ch.IOUtil.write(IOUtil.java:60)", "sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)", "org.jruby.ext.openssl.SSLSocket.writeToChannel(SSLSocket.java:485)", "org.jruby.ext.openssl.SSLSocket.flushData(SSLSocket.java:470)", "org.jruby.ext.openssl.SSLSocket.write(SSLSocket.java:514)", "org.jruby.ext.openssl.SSLSocket.do_syswrite(SSLSocket.java:697)", "org.jruby.ext.openssl.SSLSocket.syswrite(SSLSocket.java:711)", "org.jruby.ext.openssl.SSLSocket$INVOKER$i$1$0$syswrite.call(SSLSocket$INVOKER$i$1$0$syswrite.gen)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)", "rubyjit.Lumberjack::Socket$$write_b7fdc094b05c995941e5ecbb08f551c69e4682b31442407170.__file__(/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.22/lib/lumberjack/client.rb:102)", "rubyjit.Lumberjack::Socket$$write_b7fdc094b05c995941e5ecbb08f551c69e4682b31442407170.__file__(/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.22/lib/lumberjack/client.rb)", "org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:181)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)", "rubyjit.Lumberjack::Socket$$write_hash_9e7d6051ec239f7b989614b456fe2c9408333a1f1442407170.__file__(/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.22/lib/lumberjack/client.rb:110)", "rubyjit.Lumberjack::Socket$$write_hash_9e7d6051ec239f7b989614b456fe2c9408333a1f1442407170.__file__(/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.22/lib/lumberjack/client.rb)", "org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:181)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)", "rubyjit.Lumberjack::Client$$write_20ca96f87ec6c1099e37fd97b7c6c2fe94f243051442407170.__file__(/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.22/lib/lumberjack/client.rb:44)", "rubyjit.Lumberjack::Client$$write_20ca96f87ec6c1099e37fd97b7c6c2fe94f243051442407170.__file__(/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.22/lib/lumberjack/client.rb)", "org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:181)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)", "org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)", "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)", "org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)", "org.jruby.ast.RescueNode.interpret(RescueNode.java:116)", "org.jruby.ast.BeginNode.interpret(BeginNode.java:83)", "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)", "org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)", "org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)", "org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)", "org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)", "org.jruby.runtime.Block.call(Block.java:101)", "org.jruby.RubyProc.call(RubyProc.java:290)", "org.jruby.RubyProc.call19(RubyProc.java:271)", "org.jruby.RubyProc$INVOKER$i$0$0$call19.call(RubyProc$INVOKER$i$0$0$call19.gen)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:218)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:214)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:202)", "rubyjit.LogStash::Codecs::JSON$$encode_3bf46dd14047451adc96d4fa2cf3d6c75de874b11442407170.__file__(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-json-1.0.0/lib/logstash/codecs/json.rb:49)", "rubyjit.LogStash::Codecs::JSON$$encode_3bf46dd14047451adc96d4fa2cf3d6c75de874b11442407170.__file__(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-json-1.0.0/lib/logstash/codecs/json.rb)", "org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:181)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)", "rubyjit.LogStash::Outputs::Lumberjack$$receive_b8c11acaabf22ad554df0434aff24bda68c4990a1442407170.__file__(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-lumberjack-1.0.0/lib/logstash/outputs/lumberjack.rb:41)", "rubyjit.LogStash::Outputs::Lumberjack$$receive_b8c11acaabf22ad554df0434aff24bda68c4990a1442407170.__file__(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-lumberjack-1.0.0/lib/logstash/outputs/lumberjack.rb)", "org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:181)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)", "rubyjit.LogStash::Outputs::Base$$handle_55d11b738524f0d354ebf54aa7334e2156dbaabb1442407170.__file__(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/outputs/base.rb:88)", "rubyjit.LogStash::Outputs::Base$$handle_55d11b738524f0d354ebf54aa7334e2156dbaabb1442407170.__file__(/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.2.2-java/lib/logstash/outputs/base.rb)", "org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:181)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)", "org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)", "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)", "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)", "org.jruby.ast.WhileNode.interpret(WhileNode.java:131)", "org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)", "org.jruby.ast.BlockNode.interpret(BlockNode.java:71)", "org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)", "org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)", "org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)", "org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)", "org.jruby.runtime.Block.call(Block.java:101)", "org.jruby.RubyProc.call(RubyProc.java:290)", "org.jruby.RubyProc.call(RubyProc.java:228)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)", "java.lang.Thread.run(Thread.java:745)"], :level=>:error}

My output config is quite simple as you can see

output {
  if "_grokparsefailure" not in [tags] and "metric" not in [tags] {
    lumberjack {
      codec => json { }
      hosts => ["aaa.bbb.ccc.ddd"]
      port => "5043"
      ssl_certificate => "/opt/logstash/ssl/logstash.crt"
      workers => 10
    }
  }
}

I can't see any error on the other logstash server running the lumberjack input and this only appears to happen with logstash 1.5

There's one thing I've noticed and it's that the error goes away when you run the output with a single worker instead of 10. But I need the extra workers, otherwise the output becomes a bottleneck.

dsolsona commented 9 years ago

Hi,

Is there any update on this?

dsolsona commented 9 years ago

Adding more information based on feedback I got from https://github.com/elastic/logstash/issues/3559

I can easily reproduce this issue with 1.5.2-1 and two easy configuration files.

Start one logstash instance with the following configuration

input {
  lumberjack {
  codec => json {}
    port => "5001"
    ssl_certificate => "/opt/logstash/ssl/cert.crt"
    ssl_key => "/opt/logstash/ssl/cert.key"
  }
}

output { stdout { codec => rubydebug} }

Then start another logstash process with this config

input {
  generator {
    lines => "127.0.0.1 127.0.0.1 - - [10/Aug/2015:07:26:32 -0500]  GET /_fpm_status?json HTTP/1.1 200 317 - collectd/5.4.0 0.000"
    count => 100
  }
}

output {
  lumberjack {
    codec => json { }
    hosts => "127.0.0.1"
    port => "5001"
    ssl_certificate => "/opt/logstash/ssl/cert.crt"
    workers => 2
  }
}

This consistently throws the error I posted before. The same configuration without the workers => 2 works fine.

dsolsona commented 9 years ago

This is also happening with logstash 1.5.3-1 as well.

dsolsona commented 9 years ago

Any update on this? It's been more three months with no updates whatsoever :(

Kvrepo commented 5 years ago

I would like to bump this issue. I get the same error in the logs. I'm following the instructions written here: https://www.elastic.co/guide/en/logstash/6.7/ls-to-ls.html

#Input pipeline:
/usr/share/logstash/bin/logstash -e 'input{stdin{}}output{ lumberjack { codec => json ssl_certificate => "lumberjack.cert" port => 5001 hosts => "localhost" } }'

#Output pipeline
/usr/share/logstash/bin/logstash -e 'input { beats { codec => json port => 5001 ssl => true ssl_certificate => "lumberjack.cert" ssl_key => "lumberjack.key"} }output{stdout{}}'

The first few messages gets through swiftly, then the next few gets halted. Then the error mentioned above appears in the logs and suddenly all halted messages gets to the other side.