Netflow codec crashes when receiving Juniper IPFIX Port Block Allocation template

[zimage]

I have a Juniper MX104 router running Junos 16.1R6 and it is configured to send NAT port block allocation records via IPFIX. Scrutinizer handles them fine, but the logstash netflow codec dies. with the following. The ipfix template has two fields with the same name. This doesn't appear to be against any part of the RFC. Should I submit a pcap?

[2018-05-08T14:04:13,858][ERROR][logstash.inputs.udp ] Exception in inputworker {"exception"=>#<NameError: field 'observationTimeMilliseconds' in BinData::Struct, is defined multiple times.>, "backtrace"=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.2/lib/bindata/struct.rb:409:in block in ensure_field_names_are_valid'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.2/lib/bindata/struct.rb:399:in ensure_field_names_are_valid'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.2/lib/bindata/struct.rb:375:inblock in sanitize_fields'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.2/lib/bindata/sanitize.rb:266:in block in sanitize_fields'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.2/lib/bindata/sanitize.rb:283:insanitize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.2/lib/bindata/sanitize.rb:264:in sanitize_fields'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.2/lib/bindata/struct.rb:369:insanitize_fields'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.2/lib/bindata/struct.rb:345:in sanitize_parameters!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.2/lib/bindata/sanitize.rb:302:insanitize!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.2/lib/bindata/sanitize.rb:210:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.2/lib/bindata/sanitize.rb:192:insanitize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.2/lib/bindata/base.rb:302:in extract_args'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.2/lib/bindata/base.rb:249:inextract_args'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.2/lib/bindata/base.rb:81:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.2/lib/bindata/warnings.rb:21:ininitialize_with_warning'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-3.11.2/lib/logstash/codecs/netflow.rb:330:in block in decode_ipfix'", "org/jruby/ext/thread/Mutex.java:148:insynchronize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-3.11.2/lib/logstash/codecs/netflow.rb:329:in block in decode_ipfix'", "org/jruby/RubyKernel.java:1114:incatch'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-3.11.2/lib/logstash/codecs/netflow.rb:313:in block in decode_ipfix'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.2/lib/bindata/array.rb:208:inblock in each'", "org/jruby/RubyArray.java:1734:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.2/lib/bindata/array.rb:208:ineach'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-3.11.2/lib/logstash/codecs/netflow.rb:312:in decode_ipfix'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-3.11.2/lib/logstash/codecs/netflow.rb:127:inblock in decode'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.2/lib/bindata/array.rb:208:in block in each'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.2/lib/bindata/array.rb:208:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-3.11.2/lib/logstash/codecs/netflow.rb:126:indecode'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.2.1/lib/logstash/inputs/udp.rb:133:in inputworker'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.2.1/lib/logstash/inputs/udp.rb:102:inblock in udp_listener'"]}

[zimage]

Looks like this one, although it isn't about Cisco HSL, is another example of the duplicate bug mention in #93

[jorritfolmer]

Yep we can't handle duplicate fields because the BinData library we use doesn't support them. I don't see an easy fix really.

[zimage]

Would it be easy to add a suffix to the field name when parsing the template before sending the field names to BinData? This seems to be what scrutinizer does. The first "observationTimeMiliseconds" stays as is and the second one is called "observationTimeMilliseconds_v001".

[jorritfolmer]

The issue as I see it is that it breaks while parsing the template, so we don't even get far enough to receive field names.

Thanks for the suggestion though, I'll have to play some more with that part of the code to get a better understanding of the available paths forward.