search

logstash-plugins / logstash-codec-netflow

Apache License 2.0
78 stars 87 forks source link

IPFIX multiple identical fields (Was: Can't decode flowset id 258 from observation domain id 256) #169

Open AshHaque opened 6 years ago

AshHaque commented 6 years ago

For IPFIX exporter (Cisco router of 4321 model and IOS 16), I am getting this message. I run the flow for hours. But this message is not going away. Using elastiflow on top this codec.

Netflow version 9 is working fine. Problem is only with IPFIX.

logstash version : 6.4 logstash-codec-netflow: 4.2

I am new in ELK. Help will be appreciated. I attached a PCAP file if it helps.

colopcap.zip

AshHaque commented 6 years ago

When this pcap was taken I was getting error message with flowset id 257.

AshHaque commented 6 years ago

here's the latest pcap from logstash.

colo_3010.zip

AshHaque commented 6 years ago

This is the debug log:

[2018-10-30T16:15:43,884][ERROR][logstash.inputs.udp ] Exception in inputworker {"exception"=>#<NameError: field 'ciscoAppHTTPHost' in BinData::Struct, is defined multiple times.>, "backtrace"=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/struct.rb:409:in block in ensure_field_names_are_valid'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/struct.rb:399:in ensure_field_names_are_valid'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/struct.rb:375:inblock in sanitize_fields'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/sanitize.rb:266:in block in sanitize_fields'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/sanitize.rb:283:insanitize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/sanitize.rb:264:in sanitize_fields'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/struct.rb:369:insanitize_fields'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/struct.rb:345:in sanitize_parameters!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/sanitize.rb:302:insanitize!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/sanitize.rb:210:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/sanitize.rb:192:insanitize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/base.rb:302:in extract_args'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/base.rb:249:inextract_args'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/base.rb:81:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/warnings.rb:21:ininitialize_with_warning'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-4.1.2/lib/logstash/codecs/netflow.rb:603:in do_register'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-4.1.2/lib/logstash/codecs/netflow.rb:569:inblock in register'", "org/jruby/ext/thread/Mutex.java:148:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-4.1.2/lib/logstash/codecs/netflow.rb:568:inregister'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-4.1.2/lib/logstash/codecs/netflow.rb:306:in block in decode_ipfix'", "org/jruby/RubyKernel.java:1114:incatch'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-4.1.2/lib/logstash/codecs/netflow.rb:290:in block in decode_ipfix'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/array.rb:208:inblock in each'", "org/jruby/RubyArray.java:1734:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/array.rb:208:ineach'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-4.1.2/lib/logstash/codecs/netflow.rb:289:in decode_ipfix'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-4.1.2/lib/logstash/codecs/netflow.rb:105:inblock in decode'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/array.rb:208:in block in each'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/array.rb:208:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-4.1.2/lib/logstash/codecs/netflow.rb:104:indecode'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:151:in inputworker'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:63:inblock in run'"]}

AshHaque commented 6 years ago

In a single flowset Logstash is getting type 12235 (ciscoAppHTTPHost) multiple times. I thing this is the problem.

How to fix this?

jorritfolmer commented 6 years ago

There is no easy fix. The library we use to parse doesn't support multiple identical fields. Similar issues for reference: #93 #142

AshHaque commented 6 years ago

Thanks for the update. Apart from this issue my setup is running fantastic. Waiting for the fix to play with IPFIX. Just asking if there is any work in progress on it?

jorritfolmer commented 6 years ago

No progress, sorry.

dmittendorf commented 5 years ago

@jorritfolmer I ran into this same issue when trying to use OpenVSwitch as an IPFIX source, since it duplicates the interfaceName fields.

I have a working patch that addresses this problem by pre-processing the fields in the template received from the source and "hides" the duplicate/identical fields by replacing the field name with an empty string before constructing the BinData::Struct from the template fields. This allows templates with duplicate fields to be successfully processed/loaded, however, the side affect is that duplicate values received from the source will be ignored and won't be passed through in the generated events.

This seems like a reasonable trade-off, and the code change to support this is very small.

If you think this is a reasonable approach, I'll go ahead and create supporting tests and a PR for this change.

jorritfolmer commented 5 years ago

Yes that sounds like an improvement over the current state. It doesn't get us towards IPFIX RFC compliance, see #83, because there it states in chapter 8:

Collecting Processes MUST properly handle Templates with multiple identical Information Elements.

I'm no longer maintaining logstash-codec-netflow through, but I would suggest you create a PR and go from there.

ramrode commented 5 years ago

am facing the same issue as @dmittendorf and looking for a solution .

@dmittendorf can you please share your solution ?