Ruby Exception appearing randomly in Logs, crashing the UDP socket

debugloop commented 7 years ago

Version: logstash 5.2.1, logstash-input-udp 3.1.0
Operating System: CentOS 7
Config File (if you have sensitive info, please remove it):
```
input {
udp {
    port => 9060
    codec => netflow
}
}
```
Sample Data: Netflow Data from Cisco ASR9k
Steps to Reproduce: Let it run for weeks, sometimes longer.

The Log file looks as follows. There are many more of the first lines, they suddenly start and go on for about 40 minutes, then the last line appears and no more input is accepted (socket closed). Even though only one worker appears to have crashed.

logstash[5616]: 14:42:18.838 [[main]>worker2] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Retrying individual actions
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:43:29.512 [[main]>worker5] ERROR logstash.outputs.elasticsearch - Action
logstash[5616]: 14:44:10.525 [<udp.1] ERROR logstash.inputs.udp - Exception in inputworker {"exception"=>java.lang.UnsupportedOperationException: BUG: invoking UndefinedMethod.call; report at http://bugs.jruby.org, "bac
logstash[5616]: , "rubyjit.BinData::Primitive$$read_and_return_value_52eb13a4514fc2255f95e465e84e9b000b8828741554874502.__file__(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.3.5/lib/bindata/primitive.rb:11
logstash[5616]: CallSite.callBlock(CachingCallSite.java:143)", "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)", "rubyjit.BinData::Struct$$do_read_c2d32edc39a08156002b403527a4be6c0cc9de471
logstash[5616]: b)", "org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:181)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:326)", "org.jruby.runtime.callsite.Cachin
logstash[5616]: gstash/vendor/bundle/jruby/1.9/gems/bindata-2.3.5/lib/bindata/base.rb)", "org.jruby.internal.runtime.methods.JittedMethod.call(JittedMethod.java:181)", "org.jruby.runtime.callsite.CachingCallSite.cacheAn
logstash[5616]: sh/vendor/bundle/jruby/1.9/gems/bindata-2.3.5/lib/bindata/array.rb:208)",  only v only via
ia
"rubyjit$BinData::Array$$each_1d3992484ce335e724b186e0060000eb4dd6be411554874502$block_0$RUBY$__file__.call(rubyjit$BinData::Arra
logstash[5616]: 1.9/gems/logstash-codec-netflow-3.3.0/lib/logstash/codecs/netflow.rb)", "org.jruby.ast.executable.AbstractScript.__file__(AbstractScript.java:42)", "org.jruby.internal.runtime.methods.JittedMethod.call(J

Notably, the Systemd Unit does not fail, which makes this difficult to monitor, only via the logs.

kristian-kocher commented 5 years ago

I can confirm seeing the same problem on Debian 9 (stretch) running logstash 6.4 with openjdk version "1.8.0_181" and plugin version 3.3.3

jsvd commented 5 years ago

can you try upgrading the netflow codec to see if this is still an issue?

kristian-kocher commented 5 years ago

I am using logstash-codec-netflow-4.1.0 and this appears to be the latest version at the moment.

logstash-plugins / logstash-input-udp

Ruby Exception appearing randomly in Logs, crashing the UDP socket #30