search

logstash-plugins / logstash-input-gelf

Apache License 2.0
20 stars 39 forks source link

Listener dies and fails to restart causing massive loss of logs #26

Open c10l opened 8 years ago

c10l commented 8 years ago

After the GELF input crashes it enters a loop where it restarts and dies again. It looks to me like the thread that died first failed to release the socket so any subsequent attempts to bind to that same socket cause a new crash:

{:timestamp=>"2015-12-07T10:10:53.466000+0000", :message=>"gelf listener died", :exception=>#<LogStash::Json::ParserError: Invalid UTF-
8 start byte 0xa3
 at [Source: [B@5b6b2d63; line: 1, column: 1088]>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/li
b/logstash/json.rb:41:in `jruby_load'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/json.rb:38:in `jruby_load'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-gelf-2.0.2/lib/logstash/inputs/gelf.rb:99:in `udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-gelf-2.0.2/lib/logstash/inputs/gelf.rb:63:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:180:in `inputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:174:in `start_input'"], :level=>:warn}
{:timestamp=>"2015-12-07T10:10:58.467000+0000", :message=>"Starting gelf listener", :address=>"0.0.0.0:12201", :level=>:info}
{:timestamp=>"2015-12-07T10:10:58.468000+0000", :message=>"gelf listener died", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-gelf-2.0.2/lib/logstash/inputs/gelf.rb:84:in `udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-gelf-2.0.2/lib/logstash/inputs/gelf.rb:63:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:180:in `inputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:174:in `start_input'"], :level=>:warn}
{:timestamp=>"2015-12-07T10:11:03.469000+0000", :message=>"Starting gelf listener", :address=>"0.0.0.0:12201", :level=>:info}
{:timestamp=>"2015-12-07T10:11:03.470000+0000", :message=>"gelf listener died", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-gelf-2.0.2/lib/logstash/inputs/gelf.rb:84:in `udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-gelf-2.0.2/lib/logstash/inputs/gelf.rb:63:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:180:in `inputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:174:in `start_input'"], :level=>:warn}
...

It just gets stuck on this start-crash-restart loop.

msimons commented 8 years ago

Same issue here. Is someone already into the issue?

{:timestamp=>"2015-12-24T14:17:53.175000+0100", :message=>"gelf listener died", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'Exception': was expecting ('true', 'false' or 'null')
 at [Source: [B@2c565816; line: 1, column: 11]>, :backtrace=>["/app/logging-test/logstash/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/json.rb:41:in `jruby_load'", "/app/logging-test/logstash/logstash/vendor/bundle/jruby/1.9/gems/logstash-$
{:timestamp=>"2015-12-24T14:17:58.179000+0100", :message=>"gelf listener died", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/app/logging-test/logstash/logstash/vendor/bundle/jruby/1.$
{:timestamp=>"2015-12-24T14:18:03.179000+0100", :message=>"gelf listener died", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/app/logging-test/logstash/logstash/vendor/bundle/jruby/1.$
{:timestamp=>"2015-12-24T14:18:08.180000+0100", :message=>"gelf listener died", :exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in `bind'", "/app/logging-test/logstash/logstash/vendor/bundle/jruby/1.$
{
c10l commented 8 years ago

https://github.com/logstash-plugins/logstash-input-gelf/pull/27 fixes the crash. It does not fix the fact that the listener does not restart after crashing but it will avoid the crash on the JSON parser.

alex88 commented 7 years ago

Is the restart something that has to be implemented on logstash side?

Cactusbone commented 7 years ago

PR #51 should avoid listener dying