xenoeye
Lightweight Netflow/IPFIX collector
README.ru.md - документация на русском
The documentation is mostly translated automatically using Google translator, so if you see something weird - feel free to let us know.
With this collector and Netflow you can
- See the traffic of IP networks, individual IP addresses or services
- Monitor network traffic and quickly respond to bursts
- Monitor traffic patterns, packet distribution using different Netflow fields
Key Features
Please read carefully: some items may not be suitable for you
- The project is in beta state. The collector works for us, but we cannot give any guarantee that it will work for you.
- This is not a turnkey business solution, but a collector program and several auxiliary scripts. However, with the collector you can generate almost arbitrary reports, build charts, dashboards in Grafana and run scripts when the traffic exceeds the limits.
- We use the collector to monitor our networks. We are using Netflow v9 and IPFIX, so the collector supports them.
- Netflow v5 and sFlow are also supported.
- Unlike many modern collectors, we don't use Apache Kafka, Elastic stack or anything like that. The main processing take place inside the collector itself
- The documentation contains examples of building simple reports. To build more complex ones, you need at least basic knowledge of SQL
- Collector processes data in two ways: aggregates it over periods (for reports and charts), and uses moving averages to quickly respond to bursts
- Both methods can be used individually or together. For example, if a moving average detects a threshold being exceeded, you can run a custom script and immediately enable extended statistics collection
- Collector is not very demanding on resources. It can process data and build reports even on Orange Pi (analogous to Raspberry Pi) with 4 GB of memory
- Collector kernel is written in C
- The collector has only been tested under 64-bit Linux (x64, AArch64 and Elbrus)
- We are using PostgreSQL as a storage for time series. The data aggregated by the selected Netflow fields is exported there. Aggregation occurs within the collector
- Not a very large set of Netflow fields is supported out of the box, but you can add almost any field. Fields with types "integer" (various sizes) and "address" (IPv4 and IPv6) are currently supported
- The project has a very liberal ISC license. We have no plans to make commercial or semi-commercial versions. This means that we cannot make any predictions about the future of the project. But on the other hand:
- There are no hidden or artificial restrictions
Performance
Users are usually interested in at least a rough performance estimate, so we made several tests: we recorded real Netflow traffic from different routers in pcap files and played them on the loopback interface using tcpreplay at different speeds.
Tests were run on i3-2120 CPU @ 3.30GHz.
Very roughly, you can rely on following numbers:
In debug mode, when the contents of each flow are printed to a file, it turned out about 100K flow per second per one CPU.
In a slightly closer to production mode, with two monitoring objects, two sliding windows - about 700K fps per single CPU.
These numbers are best read in a pessimistic mood:
- if you load the collector with many monitoring objects with a bunch of reports and debug printing, it can choke on 100K fps/CPU or less
- most likely 700K fps and more cannot be processed on one CPU
Scaling to multiple cores is described below in the documentation
Documentation
Plans for the future
Right now we don't plan to add new features. We look at stability, work results, try to fix bugs and make the code simpler and more understandable.