ipfix/nf9 unknown elements cause whole data to be dropped

[dbardbar]

ipfix/nf9 templates define fields and their length, but not their exact type (string, byte, etc). The type is defined in list maintained by IANA (https://www.iana.org/assignments/ipfix/ipfix.xhtml), and there's rfc51024_model.go and the ipfix.elements files in the vflow project that try to keep up with known elements.

In addition, there's support for organization level elements, which vflow has no chance to keep up with each element added by every netflow exporter out there.

Today when vflow encounters an unknown element it aborts handling of other fields, see ipfix/decoder.go, method decodeData, where it returns nonfatalerror if InfoModel has no information.

I think it is not reasonable to ignore all data if one part cannot be accurately parsed, forcing vflow users to continuously chace new elements send by the multitude of vendors in their network.

Option 1) Change the logic to skip over such elements. This can perhaps be covered by an new options flag. Option 2) Encode such unknown elements to the resulting JSON, but as do not know the type, send them as octet array. This behavior can perhaps be toggled by a new options flag.

At any rate, we can leave the ipfix.elements file, in case someone does wish to add type info for some elements, if he wishes to get those, and wants vflow to properly parse them and not get them as octetArray.

In both cases, the code in ipfix/decoder.go getDataLength method must be changed, as it today checks the field type to handle the special case of length 65535. If we do not know the type, we cannot check it. The RFC at any rate doesn't say that the special encoding of 65535 can only be used for string or octetArray.

Input from the community on this would be appreciated.

[dbardbar]

As an example of a pcap with unknown fields, attaching here small pcap with enterprise elements in it. ipfix_icmp.zip