Add web interface for monitoring

[pavel-odintsov]

Subject

[s4m4n]

You mean real-time output of fastnetmon-client ?

[waszi]

I think the web interface should be external app. Fastnetmon could implement some kind of API (REST ?) or better output plugin system which could handle IP accounting data.

I was thinking that if fastnetmon could output per IP pps/bps data to riemann (http://riemann.io/) it would allow to create live dashboarding (top talkers). Alerting etc. Riemann provides API which web frontend could use. I am currently using riemann in own infrastructure for realtime monitoring. With scripting it could easily handle multiple threshold and blackholing. Fastnetmon should focus on fast packet processing and outputing data to other apps.

[pavel-odintsov]

Thank you for detailed comments!

Integration of FastNetMon with external graphing tool is very interesting for me but is very challenging. Because in case of very big monitored networks we should push really huge data blocks (16 million of two 64 bit values for pps and speed) every few seconds.

I could export speed for specific amount of hosts which consume more than XXX pps / mbps (1000pps/5mbps) per second. This could significantly reduce load on analyzer. And could provide ability to look at top talkers on network and provide visibility of attack/network consumers.

[pavel-odintsov]

Btw, standard monitoring systems like Zabbix could graph values for strictly defined list of metrics. But for DDoS monitoring we have different "talkers" every time.

And I can't imagine correct way to graph it...

[pavel-odintsov]

I will try to explain my idea.

1 second:

11.11.11.11 10kpps / 70mbps
22.22.22.22 12kpps / 140mbps
33.33.33.33 10kpps / 70mbps

2 second:

11.11.11.11 20kpps / 100mbps
22.22.22.22 12kpps / 140mbps
55.55.55.55 5kpps / 40mbps

3 second:

11.11.11.11 20kpps / 100mbps
22.22.22.22 12kpps / 140mbps
55.55.55.55 5kpps / 40mbps
77.77.77.77 100kpps / 700mbps

In 24h period we could have thousands of spikes from different host on graph.

[waszi]

I think it would require 2 processes:

• counter process - dictionary with all configured IPs and their counters (packets, octets). It could provide API and export to other systems riemann/graphite every X seconds.
• fastnetmon - dumps active counters to counter process. IPs without activity will not have updated counters but counter process will send last value which graphite/riemann should handle this easily.

Counter process would maintain huge data structure, but exports to external systems could be filtered (by pps, bps, subnet) or could be queried by API. Scaling to millions metric could be achieved by using opentsdb cluster or similiar solution. Main problem is to store archival data. To maintain current data it will only consume RAM.

[pavel-odintsov]

Data collection could be executed on FastNetMon side, it's not resource consumption task. I could execute data collection for 30 seconds for hosts which exceed some speed. And push this data to storage for graphing.

[pavel-odintsov]

Hello, folks!

I implemented ability to export total/per host load to Graphite: https://github.com/pavel-odintsov/fastnetmon/blob/master/docs/GRAPHITE_INTEGRATION.md

[pavel-odintsov]

Web interface should have full support of FastNetMon config file, i.e. we could enable/disable some features, it should have graphite or Grafana integration.

Grafana is more suitable because it could do some magic "select graphics with top values from this bunch of graphics".

Additionally, we need flow spec interface i.e. ability to build filtering rules with mouse and nice icons)

[pavel-odintsov]

We have great community contributed UI: https://github.com/ukfast/fnm-webui

[pavel-odintsov]

Another great option is following: https://fastnetmon.mkesolutions.net/en/

[pavel-odintsov]

There is one more option https://github.com/pirmins/fnm-fsgui/