search

robcowart / elastiflow

Network flow analytics (Netflow, sFlow and IPFIX) with the Elastic Stack
Other
2.48k stars 595 forks source link

Can't (yet) decode flowset id 260 from source id 2097 #515

Closed lulianbing58 closed 4 years ago

lulianbing58 commented 4 years ago

Always show me, [2020-04-03T15:32:02,443][DEBUG][logstash.codecs.netflow ][elastiflow] Start processing data flowset 260 [2020-04-03T15:32:02,443][WARN ][logstash.codecs.netflow ][elastiflow] Can't (yet) decode flowset id 260 from source id 2097, because no template to decode it with has been received. This message will usually go away after 1 minute.

not sure what i missed

robcowart commented 4 years ago

I could be that the fields are not supported by the codec. If you can share which kind of device it is and send me a PCAP of the flow records I can take a look. Send to elastiflow@gmail.com if you don't want to post here.

lulianbing58 commented 4 years ago

The device is Cisco IOS-XR, let me capture the flow records and send you later..

bellow is my found in log. [2020-04-03T15:53:51,093][DEBUG][logstash.codecs.netflow ][elastiflow] Start processing data flowset 260 [2020-04-03T15:53:51,093][ERROR][logstash.inputs.udp ][elastiflow] Exception in inputworker {"exception"=>java.lang.NullPointerException, "backtrace"=>["org.jruby.runtime.invokedynamic.InvokeDynamicSupport.callMethodMissing(InvokeDynamicSupport.java:86)", "org.jruby.runtime.invokedynamic.MathLinker.callMethod(MathLinker.java:418)", "org.jruby.runtime.invokedynamic.MathLinker.fixnumOperatorFail(MathLinker.java:209)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$method$uint_field$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:391)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$method$netflow_field_for$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:443)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$block$decode_netflow9$3(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:175)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:477)", "org.jruby.ir.targets.YieldSite.yield(YieldSite.java:105)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$block$each$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.RubyArray.each(RubyArray.java:1800)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0$VARARGS(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:223)", "org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:230)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$block$decode_netflow9$2(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:173)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.exceptions.CatchThrow.enter(CatchThrow.java:32)", "org.jruby.RubyKernel.rbCatch19Common(RubyKernel.java:1197)", "org.jruby.RubyKernel.rbCatch19(RubyKernel.java:1193)", "org.jruby.RubyKernel$INVOKER$s$rbCatch19.call(RubyKernel$INVOKER$s$rbCatch19.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$block$decode_netflow9$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:167)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:477)", "org.jruby.ir.targets.YieldSite.yield(YieldSite.java:105)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$block$each$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.RubyArray.each(RubyArray.java:1800)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0$VARARGS(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:223)", "org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:230)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$method$decode_netflow9$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:166)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$block$decode$2(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:97)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:477)", "org.jruby.ir.targets.YieldSite.yield(YieldSite.java:105)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$block$each$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.RubyArray.each(RubyArray.java:1800)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0$VARARGS(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:223)", "org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:230)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$method$decode$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:93)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:114)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:156)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:182)", "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:191)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:337)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:92)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:204)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:191)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$block$run$2(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:63)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)", "org.jruby.runtime.Block.call(Block.java:129)", "org.jruby.RubyProc.call(RubyProc.java:295)", "org.jruby.RubyProc.call(RubyProc.java:274)", "org.jruby.RubyProc.call(RubyProc.java:270)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.lang.Thread.run(Thread.java:748)"]} [2020-04-03T15:53:51,094][WARN ][logstash.codecs.netflow ][elastiflow] Can't (yet) decode flowset id 260 from source id 2097, because no template to decode it with has been received. This message will usually go away after 1 minute.

Thanks Luke

On Apr 3, 2020, at 11:49 PM, Rob Cowart notifications@github.com wrote:

I could be that the fields are not supported by the codec. If you can share which kind of device it is and send me a PCAP of the flow records I can take a look. Send to elastiflow@gmail.com mailto:elastiflow@gmail.com if you don't want to post here.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/robcowart/elastiflow/issues/515#issuecomment-608517156, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALQRZN7EXVWSCMOPBXAYLF3RKYAQHANCNFSM4L4F3EVQ.

robcowart commented 4 years ago

What version of Logstash are you using?

lulianbing58 commented 4 years ago

My logstash version: logstash-7.5.2.rpm

On Apr 4, 2020, at 12:00 AM, Rob Cowart notifications@github.com wrote:

What version of Logstash are you using?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/robcowart/elastiflow/issues/515#issuecomment-608523089, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALQRZN7B6UUS7PU4IFCSL2LRKYB3TANCNFSM4L4F3EVQ.

robcowart commented 4 years ago

I think you are seeing a JRuby issue in Logstash versions 7.4. and 7.5, which was fixed here... https://github.com/elastic/logstash/pull/11281

You should use either 7.3 or 7.6

lulianbing58 commented 4 years ago

Thanks Rob. Let me reinstall and keep you posted.

On Apr 4, 2020, at 12:00 AM, Rob Cowart notifications@github.com wrote:

What version of Logstash are you using?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/robcowart/elastiflow/issues/515#issuecomment-608523089, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALQRZN7B6UUS7PU4IFCSL2LRKYB3TANCNFSM4L4F3EVQ.

lulianbing58 commented 4 years ago

Hi Rob, I am using 7.6.2, seems meet another problem while launch logstash, is it a problem ? [2020-04-03T16:29:32,124][INFO ][logstash.outputs.elasticsearch][elastiflow] Installing elasticsearch template to _template/elastiflow-3.5.3 [2020-04-03T16:29:32,258][INFO ][logstash.filters.geoip ][elastiflow] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-ASN.mmdb"} [2020-04-03T16:29:37,828][INFO ][logstash.filters.geoip ][elastiflow] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-City.mmdb"} [2020-04-03T16:29:37,836][INFO ][logstash.filters.geoip ][elastiflow] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-City.mmdb"} [2020-04-03T16:29:37,849][INFO ][logstash.filters.geoip ][elastiflow] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-ASN.mmdb"} [2020-04-03T16:29:43,565][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][elastiflow] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team. [2020-04-03T16:29:43,571][INFO ][logstash.javapipeline ][elastiflow] Starting pipeline {:pipeline_id=>"elastiflow", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/etc/logstash/elastiflow/conf.d/10_input_ipfix_ipv4.logstash.conf", "/etc/logstash/elastiflow/conf.d/10_input_netflow_ipv4.logstash.conf", "/etc/logstash/elastiflow/conf.d/10_input_sflow_ipv4.logstash.conf", "/etc/logstash/elastiflow/conf.d/20_filter_10_begin.logstash.conf", "/etc/logstash/elastiflow/conf.d/20_filter_20_netflow.logstash.conf", "/etc/logstash/elastiflow/conf.d/20_filter_30_ipfix.logstash.conf", "/etc/logstash/elastiflow/conf.d/20_filter_40_sflow.logstash.conf", "/etc/logstash/elastiflow/conf.d/20_filter_90_post_process.logstash.conf", "/etc/logstash/elastiflow/conf.d/30_output_10_single.logstash.conf"], :thread=>"#"} [2020-04-03T16:30:34,013][INFO ][logstash.javapipeline ][elastiflow] Pipeline started {"pipeline.id"=>"elastiflow"} [2020-04-03T16:30:34,047][INFO ][logstash.inputs.udp ][elastiflow] Starting UDP listener {:address=>"0.0.0.0:2055"} [2020-04-03T16:30:34,053][INFO ][logstash.inputs.tcp ][elastiflow] Starting tcp input listener {:address=>"0.0.0.0:4739", :ssl_enable=>"false"} [2020-04-03T16:30:34,061][INFO ][logstash.inputs.udp ][elastiflow] Starting UDP listener {:address=>"0.0.0.0:6343"} [2020-04-03T16:30:34,094][INFO ][logstash.inputs.udp ][elastiflow] Starting UDP listener {:address=>"0.0.0.0:4739"} [2020-04-03T16:30:34,175][WARN ][logstash.inputs.udp ][elastiflow] Unable to set receive_buffer_bytes to desired size. Requested 33554432 but obtained 4194304 bytes. [2020-04-03T16:30:34,175][WARN ][logstash.inputs.udp ][elastiflow] Unable to set receive_buffer_bytes to desired size. Requested 33554432 but obtained 4194304 bytes. [2020-04-03T16:30:34,179][INFO ][logstash.inputs.udp ][elastiflow] UDP listener started {:address=>"0.0.0.0:6343", :receive_buffer_bytes=>"4194304", :queue_size=>"2048"} [2020-04-03T16:30:34,179][INFO ][logstash.inputs.udp ][elastiflow] UDP listener started {:address=>"0.0.0.0:2055", :receive_buffer_bytes=>"4194304", :queue_size=>"2048"} [2020-04-03T16:30:34,191][WARN ][logstash.inputs.udp ][elastiflow] Unable to set receive_buffer_bytes to desired size. Requested 33554432 but obtained 4194304 bytes. [2020-04-03T16:30:34,201][INFO ][logstash.inputs.udp ][elastiflow] UDP listener started {:address=>"0.0.0.0:4739", :receive_buffer_bytes=>"4194304", :queue_size=>"2048"} [2020-04-03T16:30:34,286][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:elastiflow], :non_running_pipelines=>[]}

On Apr 4, 2020, at 12:10 AM, Rob Cowart notifications@github.com wrote:

I think you are seeing a JRuby issue in Logstash versions 7.4. and 7.5, which was fixed here... elastic/logstash#11281 https://github.com/elastic/logstash/pull/11281 You should use either 7.3 or 7.6

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/robcowart/elastiflow/issues/515#issuecomment-608528308, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALQRZNYJSVUEGK5EXAA7433RKYDAPANCNFSM4L4F3EVQ.

lulianbing58 commented 4 years ago

Hi Rob, this is PCAP, hope it is good for shooting, [2020-04-03T16:30:36,779][WARN ][logstash.codecs.netflow ][elastiflow] Can't (yet) decode flowset id 260 from source id 2049, because no template to decode it with has been received. This message will usually go away after 1 minute. [2020-04-03T16:30:40,899][WARN ][logstash.codecs.netflow ][elastiflow] Can't (yet) decode flowset id 260 from source id 2129, because no template to decode it with has been received. This message will usually go away after 1 minute.

On Apr 3, 2020, at 11:58 PM, lukel lulianbing@outlook.com<mailto:lulianbing@outlook.com> wrote:

The device is Cisco IOS-XR, let me capture the flow records and send you later..

bellow is my found in log. [2020-04-03T15:53:51,093][DEBUG][logstash.codecs.netflow ][elastiflow] Start processing data flowset 260 [2020-04-03T15:53:51,093][ERROR][logstash.inputs.udp ][elastiflow] Exception in inputworker {"exception"=>java.lang.NullPointerException, "backtrace"=>["org.jruby.runtime.invokedynamic.InvokeDynamicSupport.callMethodMissing(InvokeDynamicSupport.java:86)", "org.jruby.runtime.invokedynamic.MathLinker.callMethod(MathLinker.java:418)", "org.jruby.runtime.invokedynamic.MathLinker.fixnumOperatorFail(MathLinker.java:209)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$method$uint_field$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:391)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$method$netflow_field_for$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:443)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$block$decode_netflow9$3(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:175)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:477)", "org.jruby.ir.targets.YieldSite.yield(YieldSite.java:105)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$block$each$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.RubyArray.each(RubyArray.java:1800)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0$VARARGS(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:223)", "org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:230)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$block$decode_netflow9$2(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:173)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.exceptions.CatchThrow.enter(CatchThrow.java:32)", "org.jruby.RubyKernel.rbCatch19Common(RubyKernel.java:1197)", "org.jruby.RubyKernel.rbCatch19(RubyKernel.java:1193)", "org.jruby.RubyKernel$INVOKER$s$rbCatch19.call(RubyKernel$INVOKER$s$rbCatch19.gen)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$block$decode_netflow9$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:167)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:477)", "org.jruby.ir.targets.YieldSite.yield(YieldSite.java:105)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$block$each$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.RubyArray.each(RubyArray.java:1800)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0$VARARGS(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:223)", "org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:230)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$method$decode_netflow9$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:166)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$block$decode$2(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:97)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:477)", "org.jruby.ir.targets.YieldSite.yield(YieldSite.java:105)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$block$each$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.RubyArray.each(RubyArray.java:1800)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0$VARARGS(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:223)", "org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:230)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$method$decode$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:93)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:114)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:156)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:182)", "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:191)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:337)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:92)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:204)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:191)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$block$run$2(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:63)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)", "org.jruby.runtime.Block.call(Block.java:129)", "org.jruby.RubyProc.call(RubyProc.java:295)", "org.jruby.RubyProc.call(RubyProc.java:274)", "org.jruby.RubyProc.call(RubyProc.java:270)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.lang.Thread.run(Thread.java:748)"]} [2020-04-03T15:53:51,094][WARN ][logstash.codecs.netflow ][elastiflow] Can't (yet) decode flowset id 260 from source id 2097, because no template to decode it with has been received. This message will usually go away after 1 minute.

Thanks Luke

On Apr 3, 2020, at 11:49 PM, Rob Cowart notifications@github.com<mailto:notifications@github.com> wrote:

I could be that the fields are not supported by the codec. If you can share which kind of device it is and send me a PCAP of the flow records I can take a look. Send to elastiflow@gmail.commailto:elastiflow@gmail.com if you don't want to post here.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/robcowart/elastiflow/issues/515#issuecomment-608517156, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALQRZN7EXVWSCMOPBXAYLF3RKYAQHANCNFSM4L4F3EVQ.

lulianbing58 commented 4 years ago

works now with using 7.3 version. Thanks