Closed MikeFil closed 3 years ago
You may be dropping UDP packets, and thus missing flow records. Please take a look at the tuning recommendations in this post...
https://github.com/robcowart/elastiflow/issues/131#issuecomment-408211563
These recommended settings are complete. UDP packets are not dropping.
Is your device using sampling? If so you should confirm that it actually sends the sample rate in the flow records. If the sample rate is not in the flows, you will need to add the sampling interval in https://github.com/robcowart/elastiflow/blob/master/logstash/elastiflow/dictionaries/sampling_interval.yml.
Sampling is not used. Also there are deviations more snmp values. For example: snmp 80MBi/s - in elastiflow 150-160MBi/s
Flow data is never as exact for a specific point in time as SNMP, especially if flow timeouts are long. When it differs significantly, it is usually one of the above reasons. However to determine the exact cause would require more information about the network, and a PCAP of traffic and flow records.
What is suspicious to me is how flat the data is in the charts. This usually indicates dropped packets. Can you run netstat -su
twice about 60s apart on the machine where Logstash is running and share the results?
logstash:~$ netstat -su IcmpMsg: InType0: 9 InType3: 42 InType8: 13281 OutType0: 13281 OutType3: 13692 OutType8: 24 Udp: 105820002 packets received 473464 packets to unknown port received 8043502 packet receive errors 5379 packets sent 8043502 receive buffer errors 0 send buffer errors IgnoredMulti: 6 UdpLite: IpExt: InMcastPkts: 4 InBcastPkts: 6 InOctets: 183173278156 OutOctets: 3051951872410 InMcastOctets: 144 InBcastOctets: 468 InNoECTPkts: 196505153
logstash:~$ netstat -su IcmpMsg: InType0: 9 InType3: 42 InType8: 13281 OutType0: 13281 OutType3: 13692 OutType8: 24 Udp: 105824291 packets received 473464 packets to unknown port received 8043502 packet receive errors 5379 packets sent 8043502 receive buffer errors 0 send buffer errors IgnoredMulti: 6 UdpLite: IpExt: InMcastPkts: 4 InBcastPkts: 6 InOctets: 183179751003 OutOctets: 3052089155976 InMcastOctets: 144 InBcastOctets: 468 InNoECTPkts: 196512135
If you can provide a PCAP of your flow records I can take a look for anything unexpected, but if you have no logs or other indication of dropped/missed data the issue might be the device under reporting.
Unfortunately, the security service prohibits the sending of PCAP. Thank you for your help.
Hello. There is new information on this issue. Overstatement of bandwidth measurements in netflow are due to lack of flow deduplication. And incorrect display on examples occurs on cisco asr, at the moment, with the help of an engineer from cisco, the cause of this problem is sought.
Hello, Is there a progress? Graphs build upon data from my sFlow v5 agents (cisco nexus 3k) are not accurate. The accuracy is bad and there's no pattern/ratio when I compare it with my graphs gathered through snmp. sflow-sampling-interval is 4096 (this is the min, can't put it lower). Looks like logstash-sflow-codec doesn't take into account counter samples. Any thoughts how to improve that?
@MikeFil as long as only one device/exporter is being viewed, flow deduplication shouldn't make a difference. Unfortunately I am unable to further troubleshoot this with a sample of the data.
Hello. Could you help with the problem - on some hosts the bandwidth is very different from the snmp data. Also, the more interfaces on such a host the netflow-v9 is monitored, the greater the deviation in the bandwidth readings. Example:
![2](https://user-images.githubusercontent.com/15709130/81611920-9d5a7880-93e4-11ea-9f31-e4ed748164a3.png)