search

pavel-odintsov / fastnetmon

FastNetMon - very fast DDoS sensor with sFlow/Netflow/IPFIX/SPAN support
https://fastnetmon.com
GNU General Public License v2.0
3.41k stars 566 forks source link

Data inaccuracy issue with Cisco Routers? #456

Closed babarnazmi closed 8 years ago

babarnazmi commented 8 years ago

I am having data inaccuracy issue, data is not accurate with Cisco Router 7200 VXR. netflow_sampling_ratio = 1 is enabled (installed a day before yesterday with auto install perl script on CentOS 6.5, I doubt auto install is installing old version as I don't have key shortcuts in fastnetmon-client and in /tmp folder some files look different than git master, like prefix option not working for graphite )

I have total 400 Mbps of connectivity and fastnetmon triggered notify email script is showing 600Mbps of attack on single IP which is impossible as provider will not allow more than 400Mbps on my circuit. Outgoing attack is also inaccurate when I restrict bandwidth on 30 Mbps (bandwidth shaping is done before router) to single user but it reports 550 Mbps of outgoing traffic.

Secondly notify email saying attack of 600Mbps attack whereas, graphite+grafana (highestMax(fastnetmon..incoming.pps, 10)) is showing 1.6 K (in graphs, attached) and not sure about packets value (that they are accurate or not?). Note: there is no "fastnetmon..incoming.average.pps/average.mbps", as mentioned in documentation for top talkers. I am doing something wrong here? CentOS version issue etc?

Is this a known issue? How to fix it?

top-bandwidth-usage

pavel-odintsov commented 8 years ago

Hello!

Could you share your netflow active/inactive timeout? What is your average_calculation_time?

I recommend to set all of them to 30 (netflow: active/incative + average_calculation_time = 30) and you will got accurate data here :)

babarnazmi commented 8 years ago

Hi, I know it's me :+1:

Currently it is: ip flow-cache timeout active 1

I got only Router(config)#ip flow-cache timeout ? active Specify the active flow timeout inactive Specify the inactive flow timeout

How to show/set average_calculation_time?

pavel-odintsov commented 8 years ago

average_calculation_time should be specified in /etc/fastnetmon.conf:

average_calculation_time = 30

Active/inactive... Could not help here :( I haven't experience here...

babarnazmi commented 8 years ago

OK, did that changes, will test it, where can people ask questions, like compiling to the latest version, issues with auto install, etc.?

Thanks a lot for writing such a cool tool, it is so good and with Attack Triggers/Grafana, so amazing :)

pavel-odintsov commented 8 years ago

Thanks so much! :) You could subscribe to maillist: https://groups.google.com/forum/#!forum/fastnetmon

pavel-odintsov commented 8 years ago

Sorry for mistake! You need to specify:

average_calculation_time = 30
babarnazmi commented 8 years ago

Router(config)#ip flow-cache timeout active 30 & avegae_calculation_time = 30

I can use only 'active' or 'inactive' at once in Cisco, Since that changes receiving so many false positive.

Still values are so incorrect, both mbps and pps, almost 10 times showing 700K pps (TX) whereas in actual they are 700 pps (TX)

IP: 65.x.x.x Attack type: unknown Initial attack power: 42982 packets per second Peak attack power: 42982 packets per second Attack direction: incoming Attack protocol: tcp Total incoming traffic: 378 mbps Total outgoing traffic: 0 mbps Total incoming pps: 42982 packets per second Total outgoing pps: 0 packets per second Total incoming flows: 0 flows per second Total outgoing flows: 0 flows per second Average incoming traffic: 378 mbps Average outgoing traffic: 0 mbps Average incoming pps: 42982 packets per second Average outgoing pps: 0 packets per second Average incoming flows: 0 flows per second Average outgoing flows: 0 flows per second Incoming ip fragmented traffic: 0 mbps Outgoing ip fragmented traffic: 0 mbps Incoming ip fragmented pps: 0 packets per second Outgoing ip fragmented pps: 0 packets per second Incoming tcp traffic: 0 mbps Outgoing tcp traffic: 0 mbps Incoming tcp pps: 0 packets per second Outgoing tcp pps: 0 packets per second Incoming syn tcp traffic: 0 mbps Outgoing syn tcp traffic: 0 mbps Incoming syn tcp pps: 0 packets per second Outgoing syn tcp pps: 0 packets per second Incoming udp traffic: 0 mbps Outgoing udp traffic: 0 mbps Incoming udp pps: 0 packets per second Outgoing udp pps: 0 packets per second Incoming icmp traffic: 0 mbps Outgoing icmp traffic: 0 mbps Incoming icmp pps: 0 packets per second Outgoing icmp pps: 0 packets per second Average packet size for incoming traffic: 1154.8 bytes Average packet size for outgoing traffic: 0.0 bytes

pavel-odintsov commented 8 years ago

What about netflow sampling? Do you use it? We could not extract it from flow so you need to specify it with configuration param:

# Netflow v9 and IPFIX agents use different and very complex approaches for notifying about sample ratio
# Here you could specify a sampling ratio for all this agents
# For NetFLOW v5 we extract sampling ratio from packets directely and this option not used
netflow_sampling_ratio = 1

Could it help you?

babarnazmi commented 8 years ago

Already set to 1, as said in my first post netflow_sampling_ratio = 1

pavel-odintsov commented 8 years ago

Very strange... What about counters from fastnetmon_client? Are they similar to real?

babarnazmi commented 8 years ago

Yes counters are almost real, both traffic in mbps and pps, (showing 50-60 Mbps more bandwidth than I have, pps looks correct)

fastnetmon-counters

Please suggest me if it worth trying, I can install new Debian VPS, if it is related to some issue with auto_install script or CentOS version

pavel-odintsov commented 8 years ago

Looks great!

Could you try to install development version:

wget https://raw.githubusercontent.com/pavel-odintsov/fastnetmon/master/src/fastnetmon_install.pl -Ofastnetmon_install.pl 
sudo perl fastnetmon_install.pl --use-git-master

We have few fixes here for counters accuracy.

babarnazmi commented 8 years ago

WOW, Counters are now perfect, as well as values in graphite, some fields are changed, like fastnetmon.hosts.* and .mbps now changed to .bps

fastnetmon-counters2

Install PF_RING dependencies with package manager Download PF_RING 6.0.3 sources Unpack PF_RING Build PF_RING kernel module Unload PF_RING if it was installed earlier Load PF_RING module into kernel PF_RING load error! Please fix this issue manually Build PF_RING lib Create library symlink Add pf_ring to ld.so.conf Install json library Download archive Uncompress it Build it Install it Download nDPI Configure nDPI Build and install nDPI Add ndpi to ld.so.conf Download Luajit Unpack Luajit Build and install Luajit Install LUA lpeg module Download archive Install lpeg library Install LUA json module Download archive Install it Download hiredis Build hiredis Add hiredis to ld.so.conf Download mongo Build mongo client Download log4cpp sources Unpack log4cpp sources Build log4cpp Add log4cpp to ld.so.conf Install FastNetMon dependency list Clone FastNetMon repo Install fastnetmon to dir /opt/fastnetmon If you have any issues, please check /var/log/fastnetmon.log file contents Please add your subnets in /etc/networks_list in CIDR format one subnet per line We created service fastnetmon for you You could run it with command: /etc/init.d/fastnetmon start We have built project in 5.82 minutes

PPS looks perfect too fastnetmon-pps

but bps are shown in millions? it is ok? fastnetmon-bps

Rest everything looks perfect, superb, great great great

pavel-odintsov commented 8 years ago

Yep, millions is OK because Graphite do bot know about metric unit :)

babarnazmi commented 8 years ago

Thanks a lot Sir, thanks a lot for writing such a nice tool and sharing it with us, and special thanks for your kind help.

Best Regards

pavel-odintsov commented 8 years ago

Welcome :))) I will be very happy if you could offer feedback from your company for our site!

babarnazmi commented 8 years ago

It will be my pleasure to do so, I will do it in couple of days after testing. Can you please your site link?

pavel-odintsov commented 8 years ago

http://pavel-odintsov.github.io/fastnetmon/ :)

emiljoensson commented 8 years ago

I've experienced this data inaccuracy with Cisco Netflow data as well. Do you know when the improvements will be in latest stable version?

Thanks!

pavel-odintsov commented 8 years ago

Hello!

Due to this nature netflow will be inaccurate each time. But if you are keeping enough big average_calcualtion time and keep active/inactive timeouts as short as possible you could get enough well data.

pavel-odintsov commented 8 years ago

But actually current Git version has few fixes for netflow and could help you.

zengija commented 7 years ago

Hi guys. Just wanted you to know my team had the exact same problem with inaccurate data before installing this version. Our setup consists of Juniper routers with netflow stream pointed to fastnetmon on a Vmware machine.

P.S. We have a Python notification script combined with RouterOS scripts on MikroTik routers which do the bans/unbans via BGP. If someone should need a hand with that kind of a setup I can create a guide for automated blackholing.

Pavel, you are just awesome... there are so many products like this that cost tons of money but do the same, or even less than Fastnetmon. My company is a Cloud services provider with multiple 10G uplinks that are protected with your software, and it works like a charm! Keep up the good work, you made a lot of people happy with this. 👍

pavel-odintsov commented 7 years ago

@zengija thank you for the awesome feedback! :) Regarding Mikrotik boxes, we have another ticket https://github.com/pavel-odintsov/fastnetmon/issues/620 and now it blocked on Mikrotik's developers.