Open level20peon opened 5 months ago
synfinatic
commented
5 months ago Have you looked at the NetFlow spec? That's mostly what I used to map things. If I had to guess, the .XXX indicates a VLAN tag/sub-interface.... But that's just a guess.
level20peon
commented
5 months ago the .XXX is actually a IANA PEN specifically for nprobe (see https://www.ntop.org/nprobe/iana-assigned-a-pen-to-ntop/). But it turns out that this isn't actually required for sending flows to ntopng.
I ran a few tests and stripped down the amount of fields sent by nprobe to this:
%IN_BYTES %INPUT_SNMP %L4_DST_PORT %IPV4_DST_ADDR %EXPORTER_IPV4_ADDRESS %OUTPUT_SNMP %IN_PKTS %LAST_SWITCHED %FIRST_SWITCHED %SAMPLING_INTERVAL %PROTOCOL %IN_SRC_MAC %OUT_DST_MAC %SRC_VLAN %IP_PROTOCOL_VERSION %DIRECTION %L4_SRC_PORT %IPV4_SRC_ADDR
I also modified https://github.com/synfinatic/netflow2ng/blob/main/cmd/zmq.go to output exactly the same (note: omitting IPV6 here).
This is an example of a zmq message from nprobe:
{
"1": 636,
"10": 0,
"11": 80,
"12": "192.168.0.64",
"130": "192.168.0.1",
"14": 0,
"2": 9,
"21": 1706087646,
"22": 1706087642,
"34": 1,
"4": 6,
"56": "00:00:00:00:00:00",
"57": "00:00:00:00:00:00",
"58": 0,
"60": 4,
"61": 2,
"7": 53233,
"8": "192.168.0.4"
}This is an example of a zmq message from netflow2ng:
{
"1": 736,
"10": 0,
"11": 56358,
"12": "192.168.0.41",
"130": "192.168.0.1",
"14": 0,
"2": 10,
"21": 1706088876,
"22": 1706088856,
"34": 1,
"4": 6,
"56": "00:00:00:00:00:00",
"57": "00:00:00:00:00:00",
"58": 0,
"60": 4,
"61": 0,
"7": 143,
"8": "138.201.223.84"
}As you can see, the output has the same structure. Still, the nprobe messages result in actual flows in ntopng, and the netflow2ng messages don't result in flows in ntopng, only in zmq messages.
By the way, the ntopng log doesn't complain about anything (no errors, no warnings at debug log level).
Do you have any pointers on how to proceed here? Did you try to use netflow2ng with ntopng 6.0?
level20peon
commented
5 months ago Maybe this is about templates from the netflow source? I am using softflowd on openwrt and I noticed that ntopng only starts parsing flows on a specific nprobe after the flow template is sent by softflowd. It does so automatically periodically, but I can also trigger this by using softflowctl send-template. nprobe then outputs this to the syslog:
Jan 24 11:07:36 ntopng nprobelan[10028]: [collect.c:1860] Added new flow template definition [id=1024][flow_version=9][netflow_device=192.168.0.1:35268][observation_domain_id=0][total=1]
Jan 24 11:07:36 ntopng nprobelan[10028]: [collect.c:1860] Added new flow template definition [id=1025][flow_version=9][netflow_device=192.168.0.1:35268][observation_domain_id=0][total=2]
Jan 24 11:07:36 ntopng nprobelan[10028]: [collect.c:1860] Added new flow template definition [id=2048][flow_version=9][netflow_device=192.168.0.1:35268][observation_domain_id=0][total=3]
Jan 24 11:07:36 ntopng nprobelan[10028]: [collect.c:1860] Added new flow template definition [id=2049][flow_version=9][netflow_device=192.168.0.1:35268][observation_domain_id=0][total=4]
Jan 24 11:07:36 ntopng nprobelan[10028]: [collect.c:1860] Added new flow template definition [id=256][flow_version=9][netflow_device=192.168.0.1:35268][observation_domain_id=0][total=5]I don't see netflow2ng reacting to those template transmissions at all. Could this be relevant?
edit: after having a look at the /templates endpoint of the netflow2ng webserver, it seems that those template definitions are imported after all, so this shouldn't be the issue here!?
{"192.168.0.1":{"9":{"0":{"1024":{"TemplateId":1024,"FieldCount":14,"Fields":[{"Type":8,"Length":4},{"Type":12,"Length":4},{"Type":22,"Length":4},{"Type":21,"Length":4},{"Type":1,"Length":4},{"Type":2,"Length":4},{"Type":10,"Length":4},{"Type":14,"Length":4},{"Type":7,"Length":2},{"Type":11,"Length":2},{"Type":4,"Length":1},{"Type":6,"Length":1},{"Type":60,"Length":1},{"Type":5,"Length":1}]},"1025":{"TemplateId":1025,"FieldCount":11,"Fields":[{"Type":8,"Length":4},{"Type":12,"Length":4},{"Type":22,"Length":4},{"Type":21,"Length":4},{"Type":1,"Length":4},{"Type":2,"Length":4},{"Type":10,"Length":4},{"Type":14,"Length":4},{"Type":32,"Length":2},{"Type":60,"Length":1},{"Type":5,"Length":1}]},"2048":{"TemplateId":2048,"FieldCount":14,"Fields":[{"Type":27,"Length":16},{"Type":28,"Length":16},{"Type":22,"Length":4},{"Type":21,"Length":4},{"Type":1,"Length":4},{"Type":2,"Length":4},{"Type":10,"Length":4},{"Type":14,"Length":4},{"Type":7,"Length":2},{"Type":11,"Length":2},{"Type":4,"Length":1},{"Type":6,"Length":1},{"Type":60,"Length":1},{"Type":5,"Length":1}]},"2049":{"TemplateId":2049,"FieldCount":11,"Fields":[{"Type":27,"Length":16},{"Type":28,"Length":16},{"Type":22,"Length":4},{"Type":21,"Length":4},{"Type":1,"Length":4},{"Type":2,"Length":4},{"Type":10,"Length":4},{"Type":14,"Length":4},{"Type":139,"Length":2},{"Type":60,"Length":1},{"Type":5,"Length":1}]},"256":{"TemplateId":256,"ScopeLength":4,"OptionLength":8,"Scopes":[{"Type":2,"Length":4}],"Options":[{"Type":34,"Length":4},{"Type":35,"Length":1}]}}}}}
synfinatic
commented
5 months ago I haven't looked at this at all to be honest... I'm not even sure if I'm using noting 6 or 5.x off the top of my head since I very rarely even look at it anymore. Probably worth looking at the ntopng code and see what it's doing?
jamesfreeman959
commented
5 months ago Hi! I'm seeing what I think is the same issue with Ntopng 6.1 - seeing collected ZMQ messages, but no collected flows and no traffic. I turned the logging level up to 6 and buried amongst the debug messages, I did see:
No allowed interface found for zmq://x.x.x.x:5556
I've only had time to quickly review the code for Ntopng, but this statement seems to come from Ntop.cpp where there are a few functions that check if an interface is allowed. I must confess I haven't fully understood them yet. I'm not clear what the issue is yet - whether ZMQ interfaces have become a licensable feature and so are not allowed. I can't find any reference in the GUI to allowing interfaces, or in the documentation (which in turn leads me to think this might be license related but I must stress this is an assumption).
It could be a separate issue of course, but it seems rather coincidental that I'm seeing the same counter behaviour as shown in the original post, and it appears that Ntopng is deliberately dropping this traffic because the interface "isn't allowed". The Ntopng documentation does not make is clear that this is/has become a licensable feature.
ToqQrrl
commented
4 months ago I have the same problem with ntopng 6.0 (stable) ---> no traffic being recorded ... all 0
Using ntopng with --interface zmq://x.x.x.x:5556
Would love it if this was corrected :-)
gitet
commented
4 months ago I have the same problem with ntopng 6.0 (stable) ---> no traffic being recorded ... all 0
Using ntopng with --interface zmq://x.x.x.x:5556
Would love it if this was corrected :-)
Hi @ToqQrrl , I think the correct syntax to use is '--interface tcp://x.x.x.x:5556' or I may be missing something?
Anyway, with the above in my config I am able to collect flows, but if I look at the hosts, the "Throughput" and "Total Bytes" columns are both 0 for all entries.
Using netflow2ng 0.0.5 to consume netflow v9 from a opnsense firewall ntopng community edition 6.0240207 on ubuntu 22.04 (I see the same with ntopng 6.1.x running on my raspberrypi)
ToqQrrl
commented
4 months ago Hi @gitet … in the documentation it says that we should use zmq:// instead of tcp:// … but both give the same result … I have le flows but all throughput and total bytes are 0 just like you
synfinatic
commented
3 months ago I haven't had a chance yet to dig deeper, but looks like https://github.com/ntop/ntopng/blob/dev/src/ZMQParserInterface.cpp has changed significantly from 5.6 to 6.0 which most likely explains what is going on here.
synfinatic
commented
3 months ago Digging further... not sure what your problem is as I can't reproduce it using ntopng v6.0.240305.
Can you share your ntopng.conf. Are you sure your device sending the netflow packets is using v9 and not v10/IPFIX?
FWIW, looks like v5 used tcp:// and v6 is now using zmq://, but both are valid:
[--interface|-i] <interface|pcap> | Input interface name (numeric/symbolic),
| view or pcap file path, including:
| zmq://<IP address> [ZMQ flow collection]
| tcp://<IP address> [DEPRECATED ZMQ flow collection]
| kafka://<brokers list> [Kafka flow collection]
| kafka-ssl://<brokers list> [Kafka flow collection over SSL/TLS]@synfinatic Thank you for not giving up on this 👍 I'm trying to collect netflow generated by my OpnSense firewall (version 24.1.3_1) - it can do v5 or v9, I've selected v9 in the UI. Wireshark decodes the received datagrams as "Cisco Netflow/IPFix" version 9.
I'm sending the same netflow data to a raspberrypi running nfcapd - no problems over there, so the data seems to be healthy.
Netflow2ng info:
netflow2ng --version
netflow2ng v0.0.5 -- Copyright 2020-2022 Aaron Turner
c819d0568627c4eaad3982bab0c16c2630bc8ef0 (v0.0.5-7-gc819d05) built at 2024-02-13T20:51:39+0100Invoked with netflow2ng -a 192.168.10.100:2055 -z tcp://127.0.0.1:5556
Small json sample produced with -l trace added:
{"1":0,"10":2,"11":443,"12":"35.186.227.140","130":"192.168.10.254","14":1,"15":"$MY_ROUTER_IP","16":0,"17":0,"2":0,"21":1710428931,"22":1710428625,"29":24,"30":0,"32":0,"4":6,"44":24,"45":0,"5":0,"52":0,"54":0,"58":0,"59":0,"6":27,"60":4,"61":0,"7":56668,"8":"192.168.10.100","80":"00:00:00:00:00:00","81":"00:00:00:00:00:00","88":0,"89":0} My ntopng config:
grep -v '^#.*' /etc/ntopng/ntopng.conf
-G=/var/run/ntopng.pid
-i=wlp4s0
-i=tcp://127.0.0.1:5556
-w=3000
-m=192.168.10.0/24
-q
--community
-v=2BTW: I'm using the ntopng binaries provided via https://packages.ntop.org/
synfinatic
commented
3 months ago So going back to the beginning... @level20peon tried to show that nprobe is sending different data than netflow2ng. But this is expected. For example, if you look at ntop_flow.h you'll see there are four different ways the MAC address can be specified:
#define IN_SRC_MAC 56
#define OUT_DST_MAC 57
#define IN_DST_MAC 80
#define OUT_SRC_MAC 81And that makes sense since packets have a direction (in vs. out) and you have two addresses (src and dst). So the fact that two different netflow messages use different keys for the MAC addresses makes sense and if you look, both nprobe and netflow2ng are sending valid values.
That said, nothing I see in the json output of netflow2ng looks wrong or invalid. Digging through the ntopng code, I don't see anything obvious that would cause any issues and my testing has shown it works fine with both ntopng v5 and v6.
If you'd like me to investigate further, frankly I need pcaps of the netflow traffic being sent to netflow2ng and more information about how the netflow traffic being generated. @gitet mentioned using OpnSense which I assume is using softflowd? What version? what is the configuration/CLI arguments?
If it helps, here is how my pfSense box is running softflowd v1.0.0:
/usr/local/bin/softflowd -i 1:lagg0 -n 172.17.0.5:2055 -s 1 -v 9 -T full -A milli -t tcp=3600 -t tcp.rst=5 -t tcp.fin=5 -t udp=300 -p /var/run/softflowd.lagg0.pid -c /var/run/softflowd.lagg0.ctl
/usr/local/bin/softflowd -i 2:lagg0.200 -n 172.17.0.5:2055 -s 1 -v 9 -T full -A milli -t tcp=3600 -t tcp.rst=5 -t tcp.fin=5 -t udp=300 -p /var/run/softflowd.lagg0.200.pid -c /var/run/softflowd.lagg0.200.ctl
/usr/local/bin/softflowd -i 3:lagg0.400 -n 172.17.0.5:2055 -s 1 -v 9 -T full -A milli -t tcp=3600 -t tcp.rst=5 -t tcp.fin=5 -t udp=300 -p /var/run/softflowd.lagg0.400.pid -c /var/run/softflowd.lagg0.400.ctl
/usr/local/bin/softflowd -i 4:lagg0.300 -n 172.17.0.5:2055 -s 1 -v 9 -T full -A milli -t tcp=3600 -t tcp.rst=5 -t tcp.fin=5 -t udp=300 -p /var/run/softflowd.lagg0.300.pid -c /var/run/softflowd.lagg0.300.ctl
/usr/local/bin/softflowd -i 5:lagg0.500 -n 172.17.0.5:2055 -s 1 -v 9 -T full -A milli -t tcp=3600 -t tcp.rst=5 -t tcp.fin=5 -t udp=300 -p /var/run/softflowd.lagg0.500.pid -c /var/run/softflowd.lagg0.500.ctl
/usr/local/bin/softflowd -i 6:em3 -n 172.17.0.5:2055 -s 1 -v 9 -T full -A milli -t tcp=3600 -t tcp.rst=5 -t tcp.fin=5 -t udp=300 -p /var/run/softflowd.em3.pid -c /var/run/softflowd.em3.ctlFurthermore, you'll get debugging of ZMQ messages with ntopng at --verbose=5. you may also want to try not sniffing on wlp4s0 and just relying on ZMQ.
gitet
commented
3 months ago Hi @synfinatic - I dusted off and booted my old asus router which is running softflowd 1.0.0. Flow stats emitted from softflowd is received and ingested perfectly fine by my ntopng, so I may need to look closer at the stuff emitted by my opnsense box and raise an issue with the opnsense team. Btw: Opnsense is not running softflowd, they are instead using flowd version 0.9.1_5.
I've decided to just run ntopng on the opnsense box directly instead (via plugin). Works like a charm. If you're interested in a pcap containing datagrams from both of my routers, please drop me a message. If not, my part in this has come to and end for now. Sorry for leading you on a wild goose chase - I should not have assumed the problem was related to netflow2ng or ntopng.
Thanks! BR/Espen
ToqQrrl
commented
3 months ago My problem is exactly the same … opensense … sorry I couldn’t comment earlier. So you guys think the problem comes from flowd on the opensense box ?
gitet
commented
3 months ago My problem is exactly the same … opensense … sorry I couldn’t comment earlier. So you guys think the problem comes from flowd on the opensense box ?
@ToqQrrl , this is my guess, but the logic behind it is not rock solid. I've tested the following variants (all using netflow v9):
synfinatic
commented
3 months ago @gitet , no appologies necessary. We all learned something new today. :)
Anyways, the only reference for flowd that google is giving me is for a "netflow collector"- which receives NetFlow packets; not sending them. There is another package called pfflowd which seems to be written by the same person/people, but I'm unable to find the latest version of either. So no idea what OpnSense is using really.
So I'm happy to see about why this is a problem and try to fix it, but someone is going to have to provide a pcap of say 100-200 netflow v9 packets. Just attach it here to this ticket, or if you're super privacy worried about someone seeing your flow data, just email me.... my email address is my github username at gmail.
gitet
commented
3 months ago @synfinatic , happy to hear that :-) Yes, I'm jumping to conclusions - flowd is a collector. I'm not particularly fluent with FreeBSD, but as far as I am able to tell, netflow is generated by a NETGRAPH kernel module (igc0 and igc1 being my WAN and LAN interfaces):
root@firewall:/etc/rc.d # ngctl list | grep netflow
Name: ksocket_netflow_igc1 Type: ksocket ID: 000000e4 Num hooks: 1
Name: netflow_igc0 Type: netflow ID: 000000e9 Num hooks: 3
Name: ksocket_netflow_igc0 Type: ksocket ID: 000000ef Num hooks: 1
Name: netflow_igc1 Type: netflow ID: 000000de Num hooks: 3
root@firewall:/etc/rc.d # ngctl show netflow_igc0:
Name: netflow_igc0 Type: netflow ID: 000000e9 Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
export9 ksocket_netflow_igc0 ksocket 000000ef inet/dgram/udp
out1 igc0 ether 00000002 upper
iface1 igc0 ether 00000002 lower
root@firewall:/etc/rc.d # ngctl show netflow_igc1:
Name: netflow_igc1 Type: netflow ID: 000000de Num hooks: 3
Local hook Peer name Peer type Peer ID Peer hook
---------- --------- --------- ------- ---------
export9 ksocket_netflow_igc1 ksocket 000000e4 inet/dgram/udp
out2 igc1 ether 00000003 upper
iface2 igc1 ether 00000003 lower I'll generate a pcap for you and post here ASAP.
gitet
commented
3 months ago netflow.pcapng.gz @synfinatic - here you go
synfinatic
commented
3 months ago So looking at the pcaps:
35632.471.:: for ID 27 and 28, because why bother sending IPv6 addresses for IPv4 traffic...Honestly, the more I read this comment: https://github.com/synfinatic/netflow2ng/issues/52#issuecomment-1907758371 the more I think this has something to do with how you've got ntopng configured or something?
At this point, i'd suggest you enable --verbose=5 in ntopng and see if it provides any hints.
robgarrett
commented
2 months ago Any update on this issue? I see it's still open. I filed an issue #59, which was flagged as a duplicate of this issue. I'd really appreciate understanding why I am not getting any flow data in netflow2ng in my docker container. The ports are open and I can access the metrics HTTP server.
Just to confirm, with the logging set to debug, should I be seeing output on the console? After netflow2ng starts and I see the listening message, I get nothing more.
Hate to ask, but perhaps @synfinatic could attach a basic docker-compose.yml that works so we can rule out netflow2ng and then focus on ntopng and pfsense as the issue?
synfinatic
commented
2 months ago @robgarrett I closed your ticket as a duplicate because based on the limited information you provided it appeared to be a duplicate of this. If you are not getting any netflow2ng logs, then you're not sending netflow traffic to it.
If you are using pfSense/softflowd then I am incorrect and this is not the same issue, but most likely an issue with your firewall configuration.
robgarrett
commented
2 months ago If you are using pfSense/softflowd then I am incorrect and this is not the same issue, but most likely an issue with your firewall configuration.
Correct, I do sometimes see flow information in the metrics page at 0.0.0.0:8080, but nothing on the console. I also configured softflowd per your earlier comment, and ran a packet capture on the docker network and confirmed UDP traffic coming into the netflow2d container on port 2055.
Would you consider my request to create a quick docker compose? If not, does the yml file I attached to the original #59 issue work? Any help would be greatly appreciated.
synfinatic
commented
2 months ago @robgarrett your config looks fine. have you tried increasing ntopng logging? --verbose=5 will provide debugging info for the ZMQ messages.
robgarrett
commented
2 months ago metrics.txt Yup, sure did. For a while I was getting the not allowed interface error that @jamesfreeman959 reported. That was before I had all containers on the same vnet. I don't see that now, just no flows.
Going back to netflow2ng for a moment. I see the following block in the metrics, does this mean flows are making it to netflow2ng?
flow_traffic_summary_size_bytes{local_ip="10.5.0.11",local_port="2055",remote_ip="192.168.1.1",remote_port="48611",type="NetFlow",quantile="0.5"} NaN flow_traffic_summary_size_bytes{local_ip="10.5.0.11",local_port="2055",remote_ip="192.168.1.1",remote_port="48611",type="NetFlow",quantile="0.9"} NaN flow_traffic_summary_size_bytes{local_ip="10.5.0.11",local_port="2055",remote_ip="192.168.1.1",remote_port="48611",type="NetFlow",quantile="0.99"} NaN flow_traffic_summary_size_bytes_sum{local_ip="10.5.0.11",local_port="2055",remote_ip="192.168.1.1",remote_port="48611",type="NetFlow"} 692 flow_traffic_summary_size_bytes_count{local_ip="10.5.0.11",local_port="2055",remote_ip="192.168.1.1",remote_port="48611",type="NetFlow"} 1
synfinatic
commented
2 months ago @robgarrett Since you're using softflowd, i'll reopen your ticket and we can stop polluting this one.
Wouter-Lemoine
commented
1 month ago Also trying to push data to ntopng v6, and I'm getting the same problem where ntopng is showing zmq messages but 0 flows. I am using a Mikrotik board with RouterOS v7.6 to send netflow data to netflow2ng.
Running ntopng in debug mode doesn't give any extra useful information, although sometimes:
20/May/2024 15:28:22 [ZMQCollectorInterface.cpp:353] WARNING: Unsupported publisher version: is your nProbe sender outdated? [319][24][53][2][1]
I'm not sure what more info to provide but am happy to do so.
synfinatic
commented
1 month ago @GyroW what software is Mikrotik using to generate netflow data? did this work with ntopng v5?
Wouter-Lemoine
commented
1 month ago @synfinatic Mikrotik is using it's own proprietary "traffic flow". (I think).
I tried compiling ntopng v5 but I couldn't get it to work, so I don't know if it does.
synfinatic
commented
1 month ago @GyroW well this ticket is for openflow support, so maybe you should open a new issue?
I am trying to push some data into ntopng 6.0, but it doesn't receive any flows, despite receiving zmq messages:
So I went ahead and compared the output of netflow2ng (0.0.5) with the output of nprobe (10.5) and they seem to be very different: netflow2ng output
nprobe output
I am using the @NTOPNG@ template on nprobe, which expands to
%IN_SRC_MAC %OUT_DST_MAC %INPUT_SNMP %OUTPUT_SNMP %SRC_VLAN %IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV6_SRC_ADDR %IPV6_DST_ADDR %IP_PROTOCOL_VERSION %PROTOCOL %L7_PROTO %L7_CONFIDENCE %IN_BYTES %IN_PKTS %OUT_BYTES %OUT_PKTS %FIRST_SWITCHED %LAST_SWITCHED %CLIENT_TCP_FLAGS %SERVER_TCP_FLAGS %L7_PROTO_RISK %L7_RISK_SCORE %EXPORTER_IPV4_ADDRESS %DIRECTION %SAMPLING_INTERVAL %TOTAL_FLOWS_EXP %NPROBE_IPV4_ADDRESS %NPROBE_INSTANCE_NAME %FLOW_SOURCE %POST_NAT_SRC_IPV4_ADDR %POST_NAT_DST_IPV4_ADDR %POST_NAPT_SRC_TRANSPORT_PORT %POST_NAPT_DST_TRANSPORT_PORTLooking at https://github.com/cloudflare/goflow/tree/master/decoders/netflow, I can see how I could manipulate the JSON output by incorporating those additional fields into https://github.com/synfinatic/netflow2ng/blob/main/cmd/zmq.go. However, I am not sure how those "combined" fields (i.e. "35632.527") are created / what they mean. Also, I am not sure why for example the MAC addresses use different identifiers on the nprobe string (for example 56 - _IPFIX_FIELDsourceMacAddress, 57 - _IPFIX_FIELDpostDestinationMacAddress) compared to the identifiers netflow2ng uses (for example 80 - _IPFIX_FIELDdestinationMacAddress, 81 - _IPFIX_FIELDpostSourceMacAddress). Same goes for a couple of the other fields - I guess the template used by netflow2ng at the moment is an old version, compatible with previous versions of ntopng!?
So, for now at least, ntopng doesn't receive any flows from netflow2ng. Could you give any pointers on how to incorporate the changes neccessary to support a more current ntopng version?