search

phaag / nfdump

Netflow processing tools
Other
765 stars 201 forks source link

nfdump - Skip unknown record: #### type 13 #546

Closed thezoggy closed 1 month ago

thezoggy commented 1 month ago

Running version:

nfdump -V
nfdump: Version: 1.7.4-62b8fb1 Options: ZSTD BZIP2 Date: 2024-07-11 16:00:00 +0200

When looking at some traffic today I saw Skip unknown record: #### type 13 showing up. I found your previous github issue talking https://github.com/phaag/nfdump/issues/503 about this being due to firewall device doing an export, which none of the devices that send netflow to this box are.

Router is a cisco asr9k running exr 7.1.3

example:

** nfdump -M /data/nfsen/profiles-data/live/router16  -T  -r 2024/07/19/nfcapd.202407191700 -n 20 -s dstip/packets
nfdump filter:
proto tcp and src port 443 and dst net xxx.xxx.2.0/24
Skip unknown record: 52640 type 13
Skip unknown record: 52641 type 13
Skip unknown record: 79639 type 13
Skip unknown record: 79640 type 13
Skip unknown record: 204044 type 13
Skip unknown record: 204045 type 13
Top 20 Dst IP Addr ordered by packets:
Date first seen             Duration     Proto           Dst IP Addr    Flows(%)     Packets(%)       Bytes(%)         pps      bps   bpp
2024-07-19 17:00:41.625     00:00:00.000 any       xxx.xxx.2.139(BR)        1(20.0)     5000(20.0)   260000(20.0)        0        0    52
2024-07-19 17:00:48.911     00:00:00.000 any       xxx.xxx.2.164(BR)        1(20.0)     5000(20.0)   260000(20.0)        0        0    52
2024-07-19 17:00:44.088     00:00:00.000 any        xxx.xxx.2.47(BR)        1(20.0)     5000(20.0)   260000(20.0)        0        0    52
2024-07-19 17:01:03.881     00:00:00.000 any       xxx.xxx.2.177(BR)        1(20.0)     5000(20.0)   260000(20.0)        0        0    52
2024-07-19 17:01:26.049     00:00:00.000 any        xxx.xxx.2.23(BR)        1(20.0)     5000(20.0)   260000(20.0)        0        0    52
Summary: total flows: 5, total bytes: 1.3 M, total packets: 25000, avg bps: 234107, avg pps: 562, avg bpp: 52
Time window: 2024-07-19 16:58:59 - 2024-07-19 17:04:57
Total records processed: 5, passed: 5, Blocks skipped: 0, Bytes read: 47533652
Sys: 0.3251s User: 0.7437s Wall: 0.0600s flows/second: 83.3 Runtime: 0.0602s

Looking I see this shows up in each time bucket, and actual query string does not matter:

nfdump -M /data/nfsen/profiles-data/live/router16 -T -r 2024/07/19/nfcapd.202407191700 -o raw 'proto tcp' | grep Skip
Skip unknown record: 52640 type 13
Skip unknown record: 52641 type 13
Skip unknown record: 79639 type 13
Skip unknown record: 79640 type 13
Skip unknown record: 204044 type 13
Skip unknown record: 204045 type 13

nfdump -M /data/nfsen/profiles-data/live/router16 -T -r 2024/07/19/nfcapd.202407192005 -o raw 'proto gre' | grep Skip
Skip unknown record: 56362 type 13
Skip unknown record: 56363 type 13
Skip unknown record: 83628 type 13
Skip unknown record: 83629 type 13
Skip unknown record: 198003 type 13
Skip unknown record: 198004 type 13

looking at latest bucket, and looking at those specific records to try and provide some details (masking some ips for anonymity)

netflow4:/data/nfsen/profiles-data/live/router16/2024/07/19> nfdump -r nfcapd.202407192030 -o raw 'proto tcp' | grep Skip
Skip unknown record: 53717 type 13
Skip unknown record: 53718 type 13
Skip unknown record: 79892 type 13
Skip unknown record: 79893 type 13
Skip unknown record: 193006 type 13
Skip unknown record: 193007 type 13

netflow4:/data/nfsen/profiles-data/live/router16/2024/07/19> nfdump -r nfcapd.202407192030 -o raw 
...
Flow Record: 
  RecordCount  =                 89
  Flags        =               0x02 NETFLOW v9, Sampled
  Elements     =                  8: 1 2 4 7 8 12 18 36 
  size         =                156
  engine type  =                132
  engine ID    =                 32
  export sysid =                  1
  first        =      1721420966792 [2024-07-19 20:29:26.792]
  last         =      1721420975140 [2024-07-19 20:29:35.140]
  received at  =      1721421001086 [2024-07-19 20:30:01.086]
  proto        =                  6 TCP
  tcp flags    =               0x10 ...A....
  src port     =                443
  dst port     =               5499
  src tos      =                  0
  fwd status   =                 64
  in packets   =              20000
Skip unknown record: 53717 type 13
  in bytes     =           29920000
Skip unknown record: 53718 type 13
  src addr     =       23.xxx.xxx.47: AS/SG/Singapore long/lat: 1.2868/103.8503
  dst addr     =     157.xxx.xxx.141: EU/ES/Madrid long/lat: 40.4163/-3.6934
  input        =                234
  output       =                 39
  src mask     =                 24 23.xxx.xxx.0/24
  dst mask     =                 24 157.xxx.xxx.0/24
  dst tos      =                  0
  direction    =                  0
  biFlow Dir   =               0x00 
  end reason   =               0x00 
  src as       =              54113
  dst as       =              54113
  bgp next hop =       81.20.xxx.194
  ip exporter  =       xxx.xxx.1.16
  samplingID   =                  1
  pk Interval  =                  1
  sp Interval  =               4999
  ingress VRF  =         1610612736
  egress VRF   =         1610612736

Flow Record: 
  RecordCount  =                134
  Flags        =               0x02 NETFLOW v9, Sampled
  Elements     =                  8: 1 2 4 7 8 12 18 36 
  size         =                156
  engine type  =                132
  engine ID    =                 32
  export sysid =                  1
  first        =      1721420975042 [2024-07-19 20:29:35.042]
  last         =      1721420975042 [2024-07-19 20:29:35.042]
  received at  =      1721421001086 [2024-07-19 20:30:01.086]
  proto        =                  6 TCP
  tcp flags    =               0x10 ...A....
  src port     =                443
  dst port     =              62696
  src tos      =                  0
  fwd status   =                 64
  in packets   =               5000
  in bytes     =            7480000
  src addr     =       23.xxx.xxx.81: AS/SG/Singapore long/lat: 1.2868/103.8503
  dst addr     =      157.xxx.xxx.33: EU/ES/Madrid long/lat: 40.4163/-3.6934
  input        =                234
  output       =                 39
  src mask     =                 24 23.xxx.xxx.0/24
  dst mask     =                 24 157.xxx.xxx.0/24
  dst tos      =                  0
Skip unknown record: 79892 type 13
  direction    =                  0
  biFlow Dir   =               0x00 
Skip unknown record: 79893 type 13
  end reason   =               0x00 
  src as       =              54113
  dst as       =              54113
  bgp next hop =       81.20.xxx.194
  ip exporter  =       xxx.xxx.1.16
  samplingID   =                  1
  pk Interval  =                  1
  sp Interval  =               4999
  ingress VRF  =         1610612736
  egress VRF   =         1610612736

Flow Record: 
  RecordCount  =                327
  Flags        =               0x02 NETFLOW v9, Sampled
  Elements     =                  8: 1 2 4 7 8 12 18 36 
  size         =                156
  engine type  =                132
  engine ID    =                 32
  export sysid =                  1
  first        =      1721420975579 [2024-07-19 20:29:35.579]
  last         =      1721420975583 [2024-07-19 20:29:35.583]
  received at  =      1721421001086 [2024-07-19 20:30:01.086]
  proto        =                  6 TCP
  tcp flags    =               0x10 ...A....
  src port     =              26569
  dst port     =                443
  src tos      =                  0
  fwd status   =                 64
  in packets   =              10000
  in bytes     =           14320000
Skip unknown record: 193006 type 13
  src addr     =     195.xxx.xxx.13: AS/TR/Bilecik long/lat: 40.1405/29.9876
  dst addr     =    157.xxx.xxx.194: EU/ES/Barcelona long/lat: 41.3870/2.1701
Skip unknown record: 193007 type 13
  input        =                234
  output       =                249
  src mask     =                 17 195.xxx.xxx.0/17
  dst mask     =                 24 157.xxx.xxx.0/24
  dst tos      =                  0
  direction    =                  0
  biFlow Dir   =               0x00 
  end reason   =               0x00 
  src as       =               9121
  dst as       =              32934
  bgp next hop =       81.93.xxx.55
  ip exporter  =       xxx.xxx.1.16
  samplingID   =                  1
  pk Interval  =                  1
  sp Interval  =               4999
  ingress VRF  =         1610612736
  egress VRF   =         1610612736

checking some other cisco 7.1.3 devices, seeing 

netflow4:/data/nfsen/profiles-data/live/router146/2024/07/19> nfdump -r nfcapd.202407192020 -o raw 'proto tcp' | grep Skip
Skip unknown record: 20811 type 13
Skip unknown record: 20812 type 13

netflow4:/data/nfsen/profiles-data/live/router117/2024/07/19> nfdump -r nfcapd.202407192020 -o raw 'proto udp' | grep Skip
Skip unknown record: 1677 type 13
Skip unknown record: 6042 type 13

just looking around, not seeing these show up on nokia or juniper devices..

took pcap on another device and found one of these flows, screenshot from wireshark pcap-flow

phaag commented 1 month ago

Could you please send me by email one single nfcapd file, which reports this error and if possible a cap of the stream sent to the collector. I will check it.

phaag commented 1 month ago

The record is now known to the filter thread and is fixed in the master branch. It has however, no influence of the record processing and all output is correct.

thezoggy commented 1 month ago

updated and confirm I do not see the skip unknown records anymore, thanks!