Missing data / Gaps in netflow v5 data (ES 7.8)

[drolfe]

Hi All,

I seem to be getting gaps in my netflow v5 data and I"m not sure why

I have a 5 node cluster, 8 cores 24GB ram each (12gb ES JVM)

the only thing I changed from the default elastiflow is to put a redismq (netflow => logstash => redis => logstash (filters) => ES)

I have the UDP tuning done on the node receiving the netflow

see attached

Any ideas what could be causing this ?

[drolfe]

The strange thing is the netflow events seem to be very constant on the indexing rate

The message queue isn't getting backed up at all

but the flows show like they are coming in busts for some reason (140K events at 1 minute, then 0 events the next)

see attached

[robcowart]

Can you share the Logstash config for the redis output and redis input? Also, run netstat -su a couple time and check if receive buffer errors are increasing a lot.

[drolfe]

Hi Rob,

netflow receiver ` input { udp { host => "10.XX.XX.XX" port => "9995" workers => "4" queue_size => "2048" receive_buffer_bytes => "33554432" codec => netflow { versions => [5,9,10] include_flowset_id => "true" netflow_definitions => "/etc/logstash/elastiflow/definitions/netflow.yml" ipfix_definitions => "/etc/logstash/elastiflow/definitions/ipfix.yml" } type => "netflow" } }

output { redis { host => "192.168.10.9" data_type => "list" key => "netflow" } } daniel@sns-netflow:~$ `

Netstats from netflow receiver

` netstat -su IcmpMsg: InType3: 94 InType8: 59 OutType0: 59 OutType3: 16181 Udp: 4853593 packets received 6399554 packets to unknown port received 86908889 packet receive errors 86 packets sent 86908889 receive buffer errors 0 send buffer errors UdpLite: IpExt: InOctets: 113159642924 OutOctets: 73271481914 InNoECTPkts: 203142197 InECT0Pkts: 15

date Fri 02 Oct 2020 01:12:50 PM AEST

netstat -su IcmpMsg: InType3: 94 InType8: 59 OutType0: 59 OutType3: 16181 Udp: 4854645 packets received 6399554 packets to unknown port received 86952090 packet receive errors 86 packets sent 86952090 receive buffer errors 0 send buffer errors UdpLite: IpExt: InOctets: 113207865495 OutOctets: 73287364475 InNoECTPkts: 203208242 InECT0Pkts: 15 date Fri 02 Oct 2020 01:13:01 PM AEST

netstat -su IcmpMsg: InType3: 94 InType8: 59 OutType0: 59 OutType3: 16181 Udp: 4866566 packets received 6399554 packets to unknown port received 87394988 packet receive errors 86 packets sent 87394988 receive buffer errors 0 send buffer errors UdpLite: IpExt: InOctets: 113719234833 OutOctets: 73467460210 InNoECTPkts: 203923249 InECT0Pkts: 15

date Fri 02 Oct 2020 01:16:26 PM AEST `

Config example from logstash filter node

` daniel@node1:~$ cat /etc/logstash/elastiflow/conf.d/20_filter_10_begin.logstash.conf

input { redis { host => "192.168.10.9" type => "netflow" data_type => "list" key => "netflow" } }

filter {

mutate { rename => { "[host]" => "[@metadata][host]" "[type]" => "[event][dataset]" } } mutate { add_field => {........... `

[drolfe]

Was UDP buffer related I believe

This can be closed