search

path-network / logstash-codec-sflow

Logstash codec plugin to decrypt sflow
Other
35 stars 17 forks source link

Support sFlow HTTP structures #14

Open jenningsloy318 opened 5 years ago

jenningsloy318 commented 5 years ago

Hi Configured sflow to recive log from F5 LTM with elastiflow, but got folllowing error 

[2018-12-12T00:27:09,070][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-12-12T00:28:38,990][WARN ][logstash.codecs.sflow    ] Unknown record entreprise 0, format 2102
[2018-12-12T00:28:38,993][WARN ][logstash.codecs.sflow    ] Unknown record entreprise 0, format 2100
[2018-12-12T00:28:38,994][WARN ][logstash.codecs.sflow    ] Unknown record entreprise 0, format 2206
[2018-12-12T00:28:38,996][DEBUG][logstash.codecs.sflow    ] sample: {:sample_entreprise=>0, :sample_format=>1, :sample_length=>448, :sample_data=>{:flow_sequence_number=>495, :source_id_type=>3, :source_id_index=>20, :sampling_rate=>1024, :sample_pool=>503912, :drops=>0, :input_interface=>928, :output_interface=>1073741823, :record_count=>3, :records=>[{:record_entreprise=>0, :record_format=>2102, :record_length=>20, :record_data=>""}, {:record_entreprise=>0, :record_format=>2100, :record_length=>20, :record_data=>""}, {:record_entreprise=>0, :record_format=>2206, :record_length=>352, :record_data=>""}]}}
[2018-12-12T00:28:46,191][ERROR][logstash.inputs.udp      ] UDP listener died {:exception=>java.nio.channels.ClosedSelectorException, :backtrace=>["sun.nio.ch.SelectorImpl.keys(SelectorImpl.java:68)", "org.jruby.util.io.SelectorPool.put(SelectorPool.java:88)", "org.jruby.util.io.SelectExecutor.selectEnd(SelectExecutor.java:59)", "org.jruby.util.io.SelectExecutor.go(SelectExecutor.java:44)", "org.jruby.RubyIO.select(RubyIO.java:3405)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$method$udp_listener$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:121)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$method$udp_listener$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:68)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$method$run$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$inputworker$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:409)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$block$start_input$1(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:403)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call(RubyProc.java:246)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", "java.lang.Thread.run(Thread.java:748)"]}
[2018-12-12T00:28:46,197][ERROR][logstash.inputs.udp      ] UDP listener died {:exception=>java.nio.channels.ClosedSelectorException, :backtrace=>["sun.nio.ch.SelectorImpl.keys(SelectorImpl.java:68)", "org.jruby.util.io.SelectorPool.put(SelectorPool.java:88)", "org.jruby.util.io.SelectExecutor.selectEnd(SelectExecutor.java:59)", "org.jruby.util.io.SelectExecutor.go(SelectExecutor.java:44)", "org.jruby.RubyIO.select(RubyIO.java:3405)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$method$udp_listener$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:121)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$method$udp_listener$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:68)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$method$run$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$inputworker$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:409)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$block$start_input$1(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:403)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call(RubyProc.java:246)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", "java.lang.Thread.run(Thread.java:748)"]}
[2018-12-12T00:28:46,201][ERROR][org.logstash.Logstash    ] java.lang.StackOverflowError
[2018-12-12T00:28:46,201][ERROR][logstash.inputs.udp      ] UDP listener died {:exception=>java.nio.channels.ClosedSelectorException, :backtrace=>["sun.nio.ch.SelectorImpl.keys(SelectorImpl.java:68)", "org.jruby.util.io.SelectorPool.put(SelectorPool.java:88)", "org.jruby.util.io.SelectExecutor.selectEnd(SelectExecutor.java:59)", "org.jruby.util.io.SelectExecutor.go(SelectExecutor.java:44)", "org.jruby.RubyIO.select(RubyIO.java:3405)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$method$udp_listener$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:121)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$method$udp_listener$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$method$run$0(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:68)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_3_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$method$run$0$__VARARGS__(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$inputworker$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:409)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$method$inputworker$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:145)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.RUBY$block$start_input$1(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:403)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:145)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)", "org.jruby.runtime.Block.call(Block.java:124)", "org.jruby.RubyProc.call(RubyProc.java:289)", "org.jruby.RubyProc.call(RubyProc.java:246)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)", "java.lang.Thread.run(Thread.java:748)"]}

and then logstash will restart.

F5 version: 11.6.2

guilhermefigueiredo commented 5 years ago

I also have a similar error

[2019-04-09T14:13:08,821][WARN ][logstash.codecs.sflow ] Unknown record entreprise 0, format 16 [2019-04-09T14:13:08,821][WARN ][logstash.codecs.sflow ] Unknown record entreprise 0, format 0 [2019-04-09T14:13:08,822][WARN ][logstash.codecs.sflow ] Unknown record entreprise 0, format 0 [2019-04-09T14:13:08,822][WARN ][logstash.codecs.sflow ] Unknown sample entreprise 131072 - format 0 [2019-04-09T14:13:08,850][WARN ][logstash.codecs.sflow ] Invalid sflow packet received (End of file reached) [2019-04-09T14:13:08,859][WARN ][logstash.codecs.sflow ] Invalid sflow packet received (End of file reached) [2019-04-09T14:13:08,871][WARN ][logstash.codecs.sflow ] Invalid sflow packet received (integer 2969567232 too big to convert to `int')

kzemek commented 5 years ago

Records 0-2100 (extended_socket_ipv4 for flow_data, virt_node for counter_data) and 0-2102 (extended_proxy_socket_ipv4 for flow_data) are now supported.

0-2102 & 0-2106 for counter_data are both a part of sFlow HTTP structures.