[2.3.1] Multiline - ConcurrencyError

[RomanGz]

2 Logstash servers via RR DNS receiving Java stack traces via udp. After the error, it stops processing messages on that port. Port is still up & listening. We're listening on other ports as well, that are not multiline, and they continue working.

{:timestamp=>"2016-05-17T21:07:36.970000-0700", :message=>"Exception in inputworker", "exception"=>#<ConcurrencyError: Detected invalid array contents due to unsynchronized modifications with concurrent users>, "backtrace"=>["org/jruby/RubyArray.java:1147:in `<<'", "org/jruby/RubyArray.java:1168:in `push'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-multiline-2.0.11/lib/logstash/codecs/multiline.rb:204:in `buffer'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-multiline-2.0.11/lib/logstash/codecs/multiline.rb:262:in `do_previous'", "org/jruby/RubyMethod.java:124:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-multiline-2.0.11/lib/logstash/codecs/multiline.rb:198:in `decode'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-multiline-2.0.11/lib/logstash/codecs/multiline.rb:191:in `decode'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.5/lib/logstash/inputs/udp.rb:96:in `inputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.5/lib/logstash/inputs/udp.rb:73:in `udp_listener'"], :level=>:error}

input {
  udp {
    port => 9119
    type => "error_log"
    codec => multiline {
       pattern => "^<%{NONNEGINT:prifacil}>(%{TIMESTAMP_ISO8601}|%{SYSLOGTIMESTAMP})"
       negate => true
       what => previous
    }
 }
}

filter {
   if [type] == "error_log" {
     grok {
      patterns_dir => "/etc/logstash/pattern.d"
      match => {"message" => "%{ERROR_LOG}"}
      }
  }
}

ERROR_LOG <%{NONNEGINT:prifacil}>(%{TIMESTAMP_ISO8601}|%{SYSLOGTIMESTAMP} %{DATA:src_host}) \(%{WORD:source_application}\)

[RomanGz]

Seems to be stable with syslog input instead of UDP, but they all get tagged with _grokparsefailure_sysloginput now - though whatever little parsing I'm doing works.

[stania1]

We had the same issue happen to us when we were using Logstash 5.0.0.

[RomanGz]

Well. We ended up feeding our multiline errors through syslog-ng to disk & read them in.

I think UDP packets just come in out of order & intertwined with each-other. We just have syslog-ng write out a file for each hosts that logstash reads in.