Closed pwp333 closed 3 years ago
Do you have example packets?
one thing we did find is that the netflow/ipfix template is sending:
[
"uint32",
"tcpOptions"
]
but the rfc defines tcpOptions as uint64
our netflow collector also defines tcpOptions as
209:
So this is resolved?
Length of the data field is specified in template. See http://tools.ietf.org/html/rfc5101
6.2. Reduced Size Encoding of Integer and Float Types
Information Elements containing integer, string, float, and
octetarray types in the information model MAY be encoded using fewer
octets than those implied by their type in the information model
definition [RFC5102], [...]
For instance, the information model [RFC5102] defines byteCount as an
unsigned64 type, which would require 64 bits. However, if the
Exporter will never locally encounter the need to send a value larger
than 4294967295, it may chose to send the value instead as an
unsigned32. [...]
This behavior is indicated by the Exporter by specifying a type size
with a smaller length than that associated with the assigned type of
the Information Element. In the example above, the Exporter would
place a length of 4 versus 8 in the Template.
[...] The reduction in size can be to
any number of octets smaller than the original type if the data value
still fits, i.e., so that only leading zeroes are dropped. For
example, an unsigned64 can be reduced in size to 7, 6, 5, 4, 3, 2, or
1 octet(s).
Thanks for the reply. The issue we are seeing has been reported before https://github.com/logstash-plugins/logstash-codec-netflow/issues/123 and was addressed in 2018.
This is issue can be seen if using different netflow agents/exporters but we use ipt_netflow exclusively and still seeing our collector report: Template length exceeds flowset length, skipping {:template_id=>259, :template_length=>59, :record_length=>58}
This fixes the issue completely engine_id=number
Giving unique engine_id to each ipt_netflow exporter prevents template collisions at the collector and avoids flow parsing issues.
Thanks for investigating it!
We found that netflow can report wrong flow if spanning tree protocol is enabled. It looks that 1 - Spanning Tree packets are treated as IPv6 packets(HOPOPT/0) in emitted flows 2 - Sometimes flows for such packets have abnormally large IN_BYTES/OUT_BYTES 3 - Flows with truncated Application names, Wrong IP versions (0,7 or 111), Wrong direction (21, 12). 4 - Some flows end up with a negative time Some samples are attached.![image](https://user-images.githubusercontent.com/4123730/113465552-bcda0580-93e9-11eb-8fcf-fd50d158f179.png)
Could you please help to check code? Thanks.