search

phaag / nfdump

Netflow processing tools
Other
789 stars 206 forks source link

How to capture the ASN of Netflow #435

Closed rogerwh closed 1 year ago

rogerwh commented 1 year ago

Hi!

I'm recollected information of netflow with nfcapd like this:

nfcapd -D -t 60 -S 2 -p 2055 -w /tmp/netflow/2055/flows

Reading the code of the file "/src/output/output_json.c", apparently the output can show the "src_as" and "des_as", (I understand that can show de ASN of the IP) but I dont know how it works or how can I do it correctly, because the json ouput does not have the keys:

{
    "type" : "FLOW",
    "sampled" : 0,
    "export_sysid" : 1,
    "first" : "2023-04-04T14:11:44.056",
    "last" : "2023-04-04T14:11:44.056",
    "received" : "2023-04-04T14:11:59.263",
    "in_packets" : 13,
    "in_bytes" : 1870,
    "proto" : 6,
    "tcp_flags" : "......S.",
    "src_port" : 65515,
    "dst_port" : 443,
    "src_tos" : 0,
    "src4_addr" : "192.168.3.132",
    "dst4_addr" : "162.125.5.13",
    "src_geo" : "",
    "dst_geo" : "",
    "input_snmp" : 15,
    "output_snmp" : 14,
    "src_mask" : 0,
    "dst_mask" : 0,
    "src_net" : "",
    "dst_net" : "",
    "fwd_status" : 0,
    "direction" : 0,
    "dst_tos" : 0,
    "ip4_next_hop" : "201.163.91.153",
    "ip4_router" : "192.168.4.250",
    "in_src_mac" : "c8:94:02:ce:fa:df",
    "out_dst_mac" : "e4:8d:8c:7d:fd:c0",
    "in_dst_mac" : "e4:8d:8c:7d:fd:c0",
    "out_src_mac" : "e4:8d:8c:7d:fd:c1",
    "label" : "<none>"
}

What im doing wrong? I need to do something else?

rogerwh commented 1 year ago

If is posible, I can pay for consultancies, I would like that you can help me!

phaag commented 1 year ago

If your router does not export ASN information, nfdump does neither collect nor store ASN information. Typically you need a BGP peer on your router in order to include ASN information into the flow data. You would need to talk to your uostream provider, as he most likely can provide BGP. Alternatively you can use Maxmind geo information, which is built into nfdump to print ASN information. You need to configure and build nfdump with —enable-maxmind. See the geolookup man page of nfdump. It describes how to prepare the Maxmind data for nfdump.

phaag commented 1 year ago

You can sponsor the project: https://github.com/sponsors/phaag 🤗

rogerwh commented 1 year ago

Hi Peter

Sorry for my late response. You are right, I'm using Mikrotik Routeros, and don't send ASN Information to the collector. For that, I tried to use the geo flag using a maxmind database. But it gives me this error:

Open file '/home/roger/Documentos/GeoLite2-ASN_20230404/GeoLite2-ASN.mmdb': bad magic: 0x0
Error reading geo location DB file /home/roger/Documentos/GeoLite2-ASN_20230404/GeoLite2-ASN.mmdb

Maybe the error is for the file type (.mmdb) or is something that I'm missing. I will bring you up to speed

This is what I did for try nfcap and nfdump:

1.- The Router (Mikrotik) send Netflow data to port 2056 2.- In my host (ubuntu 20), I installed nfcapd clonning the repository and executing:

3.- The collector is running with this command:

nfcapd -D -t 60 -S 2 -w /tmp/netflow/2055/flows -p 2055

This generate a file like this:

├── 2023
│   └── 04
│       └── 06
│           └── 09
│               └── nfcapd.202304060951

4.- For read the binary information, use the "-o" flag:

nfdump -r 2023/04/06/09/nfcapd.202304060951 -o json > info.json

This gives me something like this:

[
{
    "type" : "FLOW",
    "sampled" : 0,
    "export_sysid" : 1,
    "first" : "2023-04-06T09:50:58.909",
    "last" : "2023-04-06T09:50:58.909",
    "received" : "2023-04-06T09:51:00.959",
    "in_packets" : 1,
    "in_bytes" : 123,
    "proto" : 6,
    "tcp_flags" : "...AP...",
    "src_port" : 33778,
    "dst_port" : 1294,
    "src_tos" : 0,
    "src4_addr" : "192.168.4.36",
    "dst4_addr" : "52.90.36.44",
    "src_geo" : "",
    "dst_geo" : "",
    "input_snmp" : 13,
    "output_snmp" : 14,
    "src_mask" : 0,
    "dst_mask" : 0,
    "src_net" : "",
    "dst_net" : "",
    "fwd_status" : 0,
    "direction" : 0,
    "dst_tos" : 0,
    "ip4_next_hop" : "201.163.91.153",
    "ip4_router" : "192.168.4.250",
    "label" : "<none>"
},
...
]

But when I try to use the "-G" flag, this error occurs:

Open file '/home/roger/Documentos/GeoLite2-ASN_20230404/GeoLite2-ASN.mmdb': bad magic: 0x0
Error reading geo location DB file /home/roger/Documentos/GeoLite2-ASN_20230404/GeoLite2-ASN.mmdb

Am I doing everything right? The .mmdb should be in other extension?

I'm testing your project, is something that would help me to much in something what I want to do. I appreciate your help and time.

Regards

phaag commented 1 year ago

Maybe you missed a step while creating the GeoDB.

You should see the country codes as well as the ASR information.

rogerwh commented 1 year ago

Thank you for answer so quickly.

I did not know that I should use .nf file. I will check it out the script that you say, and I will try it again.

Update:

Where can I find the updateGeoDB.sh file?

Update 2:

Nevermind. I found it jaja

rogerwh commented 1 year ago

Hello There!

I used th script to create an .nf file. So I executed this:

nfdump -G ~/mmdb.nf -r 2055/flows/2023/04/13/17/nfcapd.202304131744 -o json

the output:

{
    "type" : "FLOW",
    "sampled" : 0,
    "export_sysid" : 1,
    "first" : "2023-04-13T17:44:44.468",
    "last" : "2023-04-13T17:44:44.468",
    "received" : "2023-04-13T17:44:59.878",
    "in_packets" : 1,
    "in_bytes" : 73,
    "proto" : 6,
    "tcp_flags" : "...AP...",
    "src_port" : 443,
    "dst_port" : 59307,
    "src_tos" : 0,
    "src4_addr" : "52.96.103.50",
    "dst4_addr" : "192.168.3.161",
    "src_geo" : "NA/US/"San Antonio" long/lat: 29.4227/-98.4927",
    "dst_geo" : "",
    "input_snmp" : 14,
    "output_snmp" : 15,
    "src_mask" : 0,
    "dst_mask" : 0,
    "src_net" : "",
    "dst_net" : "",
    "fwd_status" : 0,
    "direction" : 0,
    "dst_tos" : 0,
    "ip4_next_hop" : "192.168.3.161",
    "ip4_router" : "192.168.4.250",
    "label" : "<none>"
}

I can see that the src_geo key has the ubication of the key src4_addr (IP), but, this dont have the ASN information. Do I need to add another flag?

Thank you for your help.

phaag commented 1 year ago

This is a bit tricky. Nfdump does not process non-existing fields. While collecting netflow data, only those fields are stored in a record which are sent from the exporter. This saves space and prevents lots of null information fields. Therefore if your router does not send AS information - even without content, no AS information exits in your records and therefore AS are not printed. There is a trick to convince nfdump, that we are interested in AS, if we apply an AS filter such as 'not src as 65536. This is a private AS and should not match a given AS unless you use that internally. This filter forces nfdump to enrich data with AS information, if a Maxmind GeoDB is supplied.

nfdump -G ~/mmdb.nf -r 2055/flows/2023/04/13/17/nfcapd.202304131744 -o json 'not src as 65535'

should print results. Please make sure to use the latest GitHub master

rogerwh commented 1 year ago

It works!

I can see the fields src_as and dst_as in the json.

{
    "type" : "FLOW",
        ...
    "dst_tos" : 0,
    "src_as" : 11172,
    "dst_as" : 14061,
        ...
},

Thank you so much for your help!

phaag commented 1 year ago

Great!

tonniteles commented 7 months ago

This is a bit tricky. Nfdump does not process non-existing fields. While collecting netflow data, only those fields are stored in a record which are sent from the exporter. This saves space and prevents lots of null information fields. Therefore if your router does not send AS information - even without content, no AS information exits in your records and therefore AS are not printed. There is a trick to convince nfdump, that we are interested in AS, if we apply an AS filter such as 'not src as 65536. This is a private AS and should not match a given AS unless you use that internally. This filter forces nfdump to enrich data with AS information, if a Maxmind GeoDB is supplied.

nfdump -G ~/mmdb.nf -r 2055/flows/2023/04/13/17/nfcapd.202304131744 -o json 'not src as 65535'

should print results. Please make sure to use the latest GitHub master

Hi Where you finded the file mmdb.nf? I only find the file on maxmind in the extension Geoip2-ASN.mmdb. This file not supported in nfdump.

thanks

phaag commented 7 months ago

you can create that file yourself. See the corresponding shell script in the maxmind directory.

tonniteles commented 7 months ago

Hi, I ran the script and also tried manually. The result is this error:

+ geolookup -G mmdb.nf 8.8.8.8
Can't process block type 3. Skip block.

Can't process block type 3. Skip block.

Can't process block type 3. Skip block.

Can't process block type 3. Skip block.

Can't process block type 3. Skip block.

Size check failed for asV4Node - rebuild nfdump geo DB
Failed to load nfdump geo DB
phaag commented 7 months ago

You habe a version miss match! Do not mix different versions of nfdump. Reinstall nfdump and try again.

tonniteles commented 7 months ago

Hi, I reinstalled the last version via github, but same error occurred :-(

geolookup -d mmdb -w mmdb.nf
Skip non file entry: ..
Skip non file entry: .
Process file: GeoLite2-City-Locations-en.csv
Loaded 88505 location records
Process file: GeoLite2-City-Blocks-IPv4.csv
Field check for GeoLite2-City-Blocks-IPv4.csv: Found extra field 'is_anycast'
Loaded 2567631 entries into IPV4 tree
Process file: GeoLite2-City-Blocks-IPv6.csv
Field check for GeoLite2-City-Blocks-IPv6.csv: Found extra field 'is_anycast'
Loaded 1151475 entries into IPV6 tree
Process file: GeoLite2-ASN-Blocks-IPv4.csv
Loaded 506334 entries into ASV4 tree
Process file: GeoLite2-ASN-Blocks-IPv6.csv
Loaded 132811 entries into ASV6 tree
+ rm -rf GeoLite2-ASN-CSV_20240402 GeoLite2-City-CSV_20240402 mmdb
+ geolookup -G mmdb.nf 8.8.8.8
Can't process block type 3. Skip block.

Can't process block type 3. Skip block.

Can't process block type 3. Skip block.

Can't process block type 3. Skip block.

Can't process block type 3. Skip block.

Size check failed for asV4Node - rebuild nfdump geo DB
Failed to load nfdump geo DB
phaag commented 7 months ago

You did not install the new compiled binaries! After make you must run make install first and then generate the new mmdd.

try

% ./geolookup -G mmdb.nf 8.8.8.8
15169   | 8.8.8.8                  | GOOGLE                           | no information | sat: 0
phaag commented 7 months ago

Please checkout again latest master, delete all old files and try again.

tonniteles commented 7 months ago

output.txt Hi, its work! But nfdump don't process AS number.

root@debian:~# nfdump -G mmdb.nf -r nfcapd.202404051500 -s dstip Top 10 Dst IP Addr ordered by flows: Date first seen Duration Proto Dst IP Addr Flows(%) Packets(%) Bytes(%) pps bps bpp 2024-04-03 23:31:27.280 00:32:10.710 any 192.141.190.67(BR) 128855( 8.0) 154625( 0.1) 12.4 M( 0.0) 80 51257 80 2024-04-03 23:29:45.240 00:33:52.750 any 132.255.92.2(BR) 42615( 2.6) 64396( 0.0) 4.3 M( 0.0) 31 16945 66 2024-04-03 23:29:10.970 00:34:27.020 any 8.8.8.8(US) 36666( 2.3) 80984( 0.1) 7.8 M( 0.0) 39 30210 96 2024-04-03 23:57:37.600 00:06:00.390 any 45.71.195.104(BR) 30491( 1.9) 35373( 0.0) 2.5 M( 0.0) 98 56217 71 2024-04-03 23:32:12.970 00:31:25.010 any 192.141.190.152(BR) 28862( 1.8) 740887( 0.5) 914.5 M( 0.7) 393 3.9 M 1234 2024-04-03 23:33:05.090 00:30:32.900 any 192.141.190.167(BR) 27378( 1.7) 706598( 0.5) 809.5 M( 0.6) 385 3.5 M 1145 2024-04-03 23:31:43.410 00:31:54.580 any 192.141.190.245(BR) 17051( 1.1) 1.0 M( 0.7) 1.3 G( 1.0) 542 5.5 M 1263 2024-04-03 23:33:26.540 00:30:11.450 any 192.141.190.221(BR) 15875( 1.0) 962840( 0.7) 1.2 G( 0.9) 531 5.4 M 1258 2024-04-03 23:32:03.400 00:31:34.590 any 8.8.4.4(US) 14095( 0.9) 43940( 0.0) 4.7 M( 0.0) 23 19636 105 2024-04-03 23:29:48.730 00:33:49.260 any 192.141.190.160(BR) 12564( 0.8) 1.7 M( 1.2) 2.1 G( 1.5) 834 8.2 M 1223 Summary: total flows: 1614039, total bytes: 134.5 G, total packets: 143.3 M, avg bps: 512.9 M, avg pps: 68295, avg bpp: 938 Time window: 2024-04-03 23:28:39 - 2024-04-04 00:03:37 Total flows processed: 1614039, passed: 1614039, Blocks skipped: 0, Bytes read: 273499604 Sys: 0.2407s User: 0.5908s Wall: 0.1434s flows/second: 11256441.5 Runtime: 0.1450s root@debian:~# nfdump -G mmdb.nf -r nfcapd.202404051500 -s dstas Top 10 Dst AS ordered by flows: Date first seen Duration Proto Dst AS Flows(%) Packets(%) Bytes(%) pps bps bpp Summary: total flows: 1614039, total bytes: 134.5 G, total packets: 143.3 M, avg bps: 512.9 M, avg pps: 68295, avg bpp: 938 Time window: 2024-04-03 23:28:39 - 2024-04-04 00:03:37 Total flows processed: 1614039, passed: 1614039, Blocks skipped: 0, Bytes read: 273499604 Sys: 0.1666s User: 0.5809s Wall: 0.0852s flows/second: 18941029.7 Runtime: 0.0853s

phaag commented 7 months ago

It's fixed in master. If you think, there is a bug, please open a new issue and do not abuse old tickets, which are already closed.