search

logstash-plugins / logstash-input-syslog

Apache License 2.0
37 stars 38 forks source link

Logstash crashes when config is reloaded and there is an open syslog connection #40

Closed oridag closed 6 years ago

oridag commented 7 years ago

I opened it here half a year ago but it probably belongs here.

When sending SIGHUP to Logstash (or using auto-reload) to reload configuration and there is an open syslog connection, Logstash crashes with the following:

[2017-08-23T08:08:24,249][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#, :backtrace=>["org/jruby/RubyIO.java:3290:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-syslog-3.2.1/lib/logstash/inputs/syslog.rb:182:intcp_receiver'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-syslog-3.2.1/lib/logstash/inputs/syslog.rb:167:in `block in tcp_listener'"]}

Evesy commented 6 years ago

Seeing a similar issue on a Logstash restart:

[2017-11-14T22:26:11,530][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<IOError: closed stream>, :backtrace=>["org/jruby/ext/socket/RubyIPSocket.java:95:in `peeraddr'", "/usr/local/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-3.2.2/lib/logstash/inputs/syslog.rb:177:in `tcp_receiver'", "/usr/local/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-3.2.2/lib/logstash/inputs/syslog.rb:167:in `tcp_listener'"]}

nerophon commented 6 years ago

The PR above likely won't be sufficient, according to @jsvd. Do we have an idea of what is required? Is simply catching the correct class of errors good enough or is the problem deeper than that?

oridag commented 6 years ago

Maybe that's not the place to discuss this, but isn't that also and issue with Logstash itself? Is it by design that an uncaught exception in a plugin causes Logstash to crash?

jordansissel commented 6 years ago

Confirmed I can reproduce this issue. #43 should improve this, but I haven't tested.

zez3 commented 3 years ago

I still get this error in LS 7.11.2 but it's GELF related

[graylog_input] Pipeline started {"pipeline.id"=>"graylog_input"}
[2021-03-23T10:29:30,352][INFO ][logstash.inputs.gelf     ][graylog_input][2df57bb99e5842e811fd30a85163a5004a744cd2197e3230fccaa71ff2191b9d] Starting gelf listener (udp) ... {:address=>"0.0.0.0:12300"}
[2021-03-23T10:29:30,359][INFO ][logstash.inputs.gelf     ][graylog_input][2df57bb99e5842e811fd30a85163a5004a744cd2197e3230fccaa71ff2191b9d] Starting gelf listener (tcp) ... {:address=>"0.0.0.0:12300"}
[2021-03-23T10:29:30,382][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :graylog_input], :non_running_pipelines=>[]}
[2021-03-23T10:29:30,499][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

I do a small pipeline change in the ES Central manager GUI and then it crashes

[2021-03-23T10:29:46,202][INFO ][logstash.pipelineaction.reload] Reloading pipeline {"pipeline.id"=>:graylog_input}
warning: thread "Ruby-0-Thread-19: :1" terminated with exception (report_on_exception is true):
IOError: stream closed in another thread
        accept at org/jruby/ext/socket/RubyTCPServer.java:161
  tcp_listener at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.0/lib/logstash/inputs/gelf.rb:127
           run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.0/lib/logstash/inputs/gelf.rb:76
[2021-03-23T10:29:46,313][INFO ][logstash.javapipeline    ][graylog_input] Pipeline terminated {"pipeline.id"=>"graylog_input"}
[2021-03-23T10:29:46,363][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<IOError: stream closed in another thread>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:161:in `accept'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.0/lib/logstash/inputs/gelf.rb:127:in `tcp_listener'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-gelf-3.3.0/lib/logstash/inputs/gelf.rb:76:in `block in run'"]}
[2021-03-23T10:29:46,375][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]

Plus if I change to the input to syslog then my ES Central manager pipeline changes are just ignored