Fastnetmon and Graphite install issue?

[RoshanaP]

**Hello, Hoping to get some help here from experts. Pardon me if my questions are too novice. I installed fastnetmon using the instructions for Debian Jessie from the following link - https://github.com/pavel-odintsov/fastnetmon/blob/master/docs/INSTALL.md

I can see from tcpdump that this server is receiving netflow packets. It shows as running as you can see below.**

root@cimmon:/# ps -ef | grep fastnetmon root 7335 1 12 11:04 ? 00:07:20 /opt/fastnetmon/fastnetmon --daemonize root 7438 1463 0 12:05 pts/1 00:00:00 grep fastnetmon

root@cimmon:/# tail -f /var/log/fastnetmon.log 2016-06-02 11:04:42,080 [INFO] Total number of monitored hosts (total size of all networks): 33793 2016-06-02 11:04:42,081 [INFO] netflow plugin started 2016-06-02 11:04:42,081 [INFO] We use custom sampling ratio for netflow: 1 2016-06-02 11:04:42,081 [INFO] netflow plugin will listen on 10.10.11.255:2055 udp port 2016-06-02 11:04:42,081 [INFO] sflow plugin started 2016-06-02 11:04:42,081 [INFO] sflow plugin will listen on 0.0.0.0:6343 udp port 2016-06-02 11:04:42,081 [INFO] Run banlist cleanup thread 2016-06-02 11:31:59,000 [INFO] Time from last run of speed_recalc is soooo big, we got ugly lags: 2 2016-06-02 11:34:58,000 [INFO] Time from last run of speed_recalc is soooo big, we got ugly lags: 2 2016-06-02 11:41:30,001 [INFO] Time from last run of speedrecalc is soooo big, we got ugly lags: 2 ^C **Now, how can I verify that it is working ? When I run /opt/fastnetmon/fastnetmonclient , I am seeing the following, not sure why it is showing 7 IPs. I thought it was supposed to show the top 7, but it seems to show some random IPs* *
root@cimmon:/# /opt/fastnetmon/fastnetmon_client FastNetMon v1.0 FastVPS Eesti OU (c) VPS and dedicated: http://FastVPS.host IPs ordered by: packets Incoming traffic 0 pps 0 mbps 0 flows 67.217.199.170 0 pps 0 mbps 0 flows 209.213.159.170 0 pps 0 mbps 0 flows 67.217.192.170 0 pps 0 mbps 0 flows 67.217.193.170 0 pps 0 mbps 0 flows 67.217.194.170 0 pps 0 mbps 0 flows 67.217.195.170 0 pps 0 mbps 0 flows 67.217.196.170 0 pps 0 mbps 0 flows

Outgoing traffic 0 pps 0 mbps 0 flows 67.217.199.170 0 pps 0 mbps 0 flows 209.213.159.170 0 pps 0 mbps 0 flows 67.217.192.170 0 pps 0 mbps 0 flows 67.217.193.170 0 pps 0 mbps 0 flows 67.217.194.170 0 pps 0 mbps 0 flows 67.217.195.170 0 pps 0 mbps 0 flows 67.217.196.170 0 pps 0 mbps 0 flows

Internal traffic 0 pps 0 mbps

Other traffic 0 pps 0 mbps

Screen updated in: 0 sec 104867 microseconds Traffic calculated in: 0 sec 42949 microseconds Total amount of not processed packets: 0

[henry-spanka]

Hi RoshanaP, no question is a bad question ;) Well, If you see netflow packets coming in then the issue lies somewhere in FastNetMon or you have an incorrect NetFlow configuration at your router/switch etc.

I see that you're running a very old version of FastNetMon so before we get into detail and debugging here I would recommend you first upgrade to the latest version and then try again to process the traffic. The newer versions also show total processed packets so if there is an issue between FastNetMon and your router it should show 0.

Follow the following steps to upgrade the toolkit:

wget https://raw.githubusercontent.com/pavel-odintsov/fastnetmon/master/src/fastnetmon_install.pl -Ofastnetmon_install.pl
chmod +x fastnetmon_install.pl
sudo perl fastnetmon_install.pl --use-git-master

Let me know your results.

[RoshanaP]

Ok, Thanks :) I installed it after stopping fastnetmon service . Then I did this - root@cimmon:/# systemctl start fastnetmon.service Warning: Unit file of fastnetmon.service changed on disk, 'systemctl daemon-reload' recommended. root@cimmon:/# systemctl daemon-reload root@cimmon:/# systemctl start fastnetmon.service root@cimmon:/# systemctl status fastnetmon.service ● fastnetmon.service - FastNetMon - DoS/DDoS analyzer with sflow/netflow/mirror support Loaded: loaded (/etc/systemd/system/fastnetmon.service; enabled) Active: active (running) since Thu 2016-06-02 15:37:56 CDT; 2min 31s ago Main PID: 7168 (fastnetmon) CGroup: /system.slice/fastnetmon.service

But, I am seeing the following now - root@cimmon:/# /opt/fastnetmon/fastnetmon_client FastNetMon 1.1.3 master git-1bfcf7d87f470b5ade455899bdf0444e2c2f29d5 FastVPS Eesti OU (c) VPS and dedicated: http://FastVPS.host IPs ordered by: packets Incoming traffic 0 pps 0 mbps 0 flows

Outgoing traffic 0 pps 0 mbps 0 flows

Internal traffic 0 pps 0 mbps

Other traffic 0 pps 0 mbps

Screen updated in: 0 sec 8271 microseconds Traffic calculated in: 0 sec 20636 microseconds Total amount of IPv6 packets related to our own network: 0 Not processed packets: 0 pps

[RoshanaP]

What is the best way to start the process ? Is doing "systemctl start fastnetmon.service" okay? One thing is, I am seeing this in error log tail -f /var/log/fastnetmon.log 2016-06-02 16:18:27,847 [ERROR] Can't store data to Graphite 2016-06-02 16:18:27,850 [ERROR] Can't store data to Graphite 2016-06-02 16:18:28,854 [ERROR] Can't store data to Graphite 2016-06-02 16:18:28,856 [ERROR] Can't store data to Graphite 2016-06-02 16:18:29,859 [ERROR] Can't store data to Graphite 2016-06-02 16:18:29,861 [ERROR] Can't store data to Graphite 2016-06-02 16:18:30,865 [ERROR] Can't store data to Graphite 2016-06-02 16:18:30,868 [ERROR] Can't store data to Graphite 2016-06-02 16:18:31,871 [ERROR] Can't store data to Graphite 2016-06-02 16:18:31,873 [ERROR] Can't store data to Graphite 2016-06-02 16:18:32,878 [ERROR] Can't store data to Graphite 2016-06-02 16:18:32,881 [ERROR] Can't store data to Graphite

[pavel-odintsov]

Hello!

Have you started carbon-cache daemon from Graphit toolkit? Looks like it's not running.

[RoshanaP]

It seems like carbon-cache is running, from below info -

root@cimmon:/# ps -ef | grep carbon-cache root 8876 1463 0 09:16 pts/1 00:00:00 grep carbon-cache _graphi+ 26552 1 0 May26 ? 00:19:11 /usr/bin/python /usr/bin/carbon-cache --config=/etc/carbon/carbon.conf --pidfile=/var/run/carbon-cache.pid --logdir=/var/log/carbon/ start

But, regardless I just turned off graphite in /etc/fastnetmon.conf just to see if fastnetmon works. After restarting the process, I don't see any graphite errors( because it is turned off) but fastnetmon is still showing 0 flows , 0 mbps ,etc , as seen below . Is it because "max_ips_in_list = 7" in the client configuration ?

FastNetMon 1.1.3 master git-1bfcf7d87f470b5ade455899bdf0444e2c2f29d5 FastVPS Eesti OU (c) VPS and dedicated: http://FastVPS.host IPs ordered by: packets Incoming traffic 0 pps 0 mbps 0 flows

Outgoing traffic 0 pps 0 mbps 0 flows

Internal traffic 0 pps 0 mbps

Other traffic 0 pps 0 mbps

Screen updated in: 0 sec 3775 microseconds Traffic calculated in: 0 sec 25231 microseconds Total amount of IPv6 packets related to our own network: 0 Not processed packets: 0 pps

[pavel-odintsov]

If you have zeros in fastnetmon_client it's not a good idea to care about graphite. Because it's optional feature and not a mandatory part of toolkit.

What's your capture traffic method? I see both sflow and netflow capture engines in configuration but actually I do not know certain capture backend name.

So then you should check with tcpdump arriving traffic information to FNM then we could investigate this issue in details.

Also I have checked you've used broadcast (.255) for sflow and I'm not sure it's correct case. FNM could have issues with broadcast addresses. Please try to replace it to unicast.

[RoshanaP]

10.10.11.255 is a /23 so it is not a broadcast address. I am only testing netflow right now. In the /etc/fastnetmon.conf I have the following -

Netflow capture method with v5, v9 and IPFIX suppotr netflow = on

Here is the tcpdump: root@cimmon:/# tcpdump -i eth0 -n -s 2000 port 2055 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 2000 bytes 10:11:11.286515 IP 216.150.xxx.xxx.62985 > 10.10.11.155.2055: UDP, length 1416 10:11:12.286333 IP 216.150.xxx.xxx.62985 > 10.10.11.155.2055: UDP, length 1416 10:11:12.286489 IP 216.150.xxx.xxx.62985 > 10.10.11.155.2055: UDP, length 1416

[pavel-odintsov]

That's very strange. It should receive this packets.

What's your active and inactive flow timeouts from netflow enabled device? What's your average_calculation_time from FastNetMon configuration?

[RoshanaP]

I didn't set any timeout value in the netflow device but I am pretty sure there is no issue there: Version 9 flow records 1428958883 flows exported in 197728201 udp datagrams 0 flows failed due to lack of export packet

In fastnetmon.conf (https://github.com/pavel-odintsov/fastnetmon/files/297911/fastnetmon.txt) I didn't change the default value of calculation time, do I need to change it to something else ?

One thing that is showing up in fastnetmon.log is 2016-06-03 09:27:23,002 [INFO] Time from last run of speed_recalc is soooo big, we got ugly lags: 2 2016-06-03 09:31:06,001 [INFO] Time from last run of speed_recalc is soooo big, we got ugly lags: 2 2016-06-03 09:53:58,001 [INFO] Time from last run of speed_recalc is soooo big, we got ugly lags: 2 2016-06-03 10:00:38,002 [INFO] Time from last run of speed_recalc is soooo big, we got ugly lags: 2 2016-06-03 10:12:28,000 [INFO] Time from last run of speed_recalc is soooo big, we got ugly lags: 2 2016-06-03 10:14:41,000 [INFO] Time from last run of speed_recalc is soooo big, we got ugly lags: 2 2016-06-03 10:45:00,005 [INFO] Time from last run of speed_recalc is soooo big, we got ugly lags: 2

[pavel-odintsov]

Hello!

Two questions: do you have iptables? Do you have rp_filter enabled: https://github.com/pavel-odintsov/fastnetmon/issues/273?

Also please try to run fastnetmon this way:

DUMP_ALL_PACKETS=yes ./fastnetmon

And it will dump all received packets to log or console. So you could see all packets as they received by FastNetMon.

Finally, FastNetMon are not receiving any packets but it should. So something wrong on Linux side here.

[RoshanaP]

Hello pavel-odintsov , Thank you for you responses. No I don't have iptables( it is a virtual server built using virtualbox) and I have a firewall in front of the server that is doing IP translation from private to public ip address. Since tcpdump is showing received packet what could be the reason that fastnetmon is not seeing it ?

I don't see any rp_filter either.

root@cimmon:/# sysctl net.ipv4.conf.default.rp_filter = 0 net.ipv4.conf.default.rp_filter = 0 sysctl: malformed setting "=" sysctl: cannot stat /proc/sys/0: No such file or directory

When I tried to run fastnetmon the way you said, I am getting the following -

root@cimmon:/# DUMP_ALL_PACKETS=yes ./fastnetmon bash: ./fastnetmon: No such file or directory

[pavel-odintsov]

With ./fastnetmon I means "full path to fastnetmon binary". I do not know full FNM path for your system, please check it. If you have installed it with installer it should be:DUMP_ALL_PACKETS=yes /opt/fastnetmon/fastnetmon

[RoshanaP]

Yes I had installed it using the installer . I did as you mentioned but logs didn't show anything other than this - 2016-06-08 08:20:00,069 [INFO] Logger initialized! 2016-06-08 08:20:00,069 [ERROR] We can't find notify script /usr/local/bin/notify_about_attack.sh 2016-06-08 08:20:00,069 [ERROR] FastNetMon is already running with pid: 8904

[pavel-odintsov]

You should stop fastnetmon before starting new copy. Just do: service fastnetmon stop

[RoshanaP]

ok, stopped and ran the command but just getting the following in the log: 2016-06-08 08:33:54,462 [INFO] Logger initialized! 2016-06-08 08:33:54,463 [ERROR] We can't find notify script /usr/local/bin/notify_about_attack.sh 2016-06-08 08:33:54,465 [INFO] Read configuration file 2016-06-08 08:33:54,466 [INFO] We are working on Linux and could use ip tool for detecting local IP's 2016-06-08 08:33:54,468 [INFO] We found 1 local IP addresses and will monitor they 2016-06-08 08:33:54,468 [INFO] We loaded 6 networks from networks file 2016-06-08 08:33:54,468 [INFO] Totally we have 7 IPv4 subnets 2016-06-08 08:33:54,468 [INFO] Totally we have 0 IPv6 subnets 2016-06-08 08:33:54,468 [INFO] Total number of monitored hosts (total size of all networks): 33793 2016-06-08 08:33:54,468 [INFO] We need 20 MB of memory for storing counters for your networks 2016-06-08 08:33:54,468 [INFO] I will allocate 1 records for subnet 2601191946 cidr mask: 32 2016-06-08 08:33:54,468 [INFO] I will allocate 8192 records for subnet 14708544 cidr mask: 19 2016-06-08 08:33:54,475 [INFO] I will allocate 4096 records for subnet 2151490 cidr mask: 20 2016-06-08 08:33:54,478 [INFO] I will allocate 4096 records for subnet 12638531 cidr mask: 20 2016-06-08 08:33:54,482 [INFO] I will allocate 1024 records for subnet 4508103 cidr mask: 22 2016-06-08 08:33:54,483 [INFO] I will allocate 8192 records for subnet 8443345 cidr mask: 19 2016-06-08 08:33:54,489 [INFO] I will allocate 8192 records for subnet 6330072 cidr mask: 19 2016-06-08 08:33:54,496 [INFO] We start total zerofication of counters 2016-06-08 08:33:54,497 [INFO] We finished zerofication 2016-06-08 08:33:54,497 [INFO] We loaded 7 IPv4 subnets to our in-memory list of networks 2016-06-08 08:33:54,497 [INFO] netflow plugin started 2016-06-08 08:33:54,498 [INFO] Using custom sampling ratio for netflow: 1 2016-06-08 08:33:54,498 [INFO] netflow: We will listen on 1 ports 2016-06-08 08:33:54,498 [INFO] sflow: plugin started 2016-06-08 08:33:54,498 [INFO] sflow: We will listen on 1 ports 2016-06-08 08:33:54,498 [INFO] Run banlist cleanup thread, we will awake every 60 seconds 2016-06-08 08:33:54,498 [INFO] netflow plugin will listen on 10.10.11.255:2055 udp port 2016-06-08 08:33:54,498 [INFO] sflow: plugin will listen on 0.0.0.0:6343 udp port

[pavel-odintsov]

As you can see FastNetMon are not receiving any data from NetFlow for some reasons. And actually I do not know why. It's Linux related issue.

[RoshanaP]

I just noticed from the tcpdump that, this server is sending icmp message back to the netflow device saying that udp port 2055 is unreachable 10:02:28.780375 IP 216.150.x.x.62985 > 10.10.11.155.2055: UDP, length 1416 10:02:30.779835 IP 216.150.x.x.62985 > 10.10.11.155.2055: UDP, length 1416 10:02:30.779866 IP 10.10.11.155 > 216.150.x.x: ICMP 10.10.11.155 udp port 2055 unreachable, length 556 10:02:30.955214 IP 216.150.x.x.62990 > 10.10.11.155.2055: UDP, length 1432 10:02:30.955237 IP 10.10.11.155 > 216.150.x.x: ICMP 10.10.11.155 udp port 2055 unreachable, length 556

Not sure why UDP port 2055 is unreachable, can we fix this ?

[pavel-odintsov]

FNM are listening 10.10.11.255 ("2016-06-08 08:33:54,498 [INFO] netflow plugin will listen on 10.10.11.255:2055 udp port").

But your device are sending to 10.10.11.155.

Just mistake in configuration. Please replace 10.10.11.255 to 10.10.11.155 in configuration file and restart FNM.

[RoshanaP]

gosh, that is a silly mistake! Thank you for noticing it :+1: I am seeing results now.

[pavel-odintsov]

Perfect!