search

CESNET / ipfixprobe

IPFIX flow exporter with DPDK support capable of bi-directional flows, per-packet-information statistics, and extensibility via processing plugins (e.g., for application layer parsers).
https://cesnet.github.io/ipfixprobe/
BSD 3-Clause "New" or "Revised" License
42 stars 18 forks source link
100gbe flow flow-exporter ipfix liberouter nemea

readme

Coverity Scan GitHub top language

ipfixprobe is a high-performance flow exporter. It creates bidirectional flows from packet input and exports them to output interface. The ipfixprobe support vide variety of flow extenstion for application layer protocol information. The flow extension can be turned on with process plugins. We support TLS, QUIC, HTTP, DNS and many more. Just check our documentation.

Installation

The RPM packages for RHEL based distributions can be downloaded from our copr repository. Or just simply run:

dnf install -y dnf-plugins-core && dnf copr -y enable @CESNET/NEMEA
dnf install ipfixprobe

Parameters

Module specific parameters

Help

Printing general help is done using the -h parameter. To print help for specific plugins, -h with parameter is used. This parameter accepts input, storage, process, output or name of a plugin (or path to a .so file with plugin).

Example

Here are the examples of various plugins usage:

# Capture from wlp2s0 interface using raw sockets, print flows to console
./ipfixprobe -i 'raw;ifc=wlp2s0' -o 'text'

# Capture from wlp2s0 interface and scale packet processing using 2 instances of plugins, send flow to ifpfix collector using UDP
./ipfixprobe -i 'raw;ifc=wlp2s0;f' -i 'raw;ifc=wlp2s0;f' -o 'ipfix;u;host=collector.example.com;port=4739'

# Capture from a COMBO card using ndp plugin, sends ipfix data to 127.0.0.1:4739 using TCP by default
./ipfixprobe -i 'ndp;dev=/dev/nfb0:0' -i 'ndp;dev=/dev/nfb0:1' -i 'ndp;dev=/dev/nfb0:2'

# Capture from eth0 interface using pcap plugin, split biflows into flows and prints them to console without mac addresses, telemetry data are exposed via the appFs library in /var/run/ipfixprobe directory
./ipfixprobe -i 'pcap;ifc=eth0' -s 'cache;split' -o 'text;m' -t /var/run/ipfixprobe

# Read packets from pcap file, enable 4 processing plugins, sends L7 HTTP extended biflows to unirec interface named `http` and data from 3 other plugins to the `stats` interface
./ipfixprobe -i 'pcap;file=pcaps/http.pcap' -p http -p pstats -p idpcontent -p phists -o 'unirec;i=u:http:timeout=WAIT,u:stats:timeout=WAIT;p=http,(pstats,phists,idpcontent)'

# Read packets using DPDK input interface and 1 DPDK queue, enable plugins for basic statistics, http and tls, output to IPFIX on a local machine
# DPDK EAL parameters are passed in `e, eal` parameters
# DPDK plugin configuration has to be specified in the first input interface.
# The following `dpdk` interfaces are given without parameters; their configuration is inherited from the first one.
# Example for the queue of 3 DPDK input plugins (q=3):
`./ipfixprobe -i "dpdk;p=0;q=3;e=-c 0x1 -a  <[domain:]bus:devid.func>" -i dpdk -i dpdk -p http "-p" bstats -p tls -o "ipfix;h=127.0.0.1"`

# Same example for the multiport read from ports 0 and 1, note comma separated ports:
`./ipfixprobe -i "dpdk;p=0,1;q=3;e=-c 0x1 -a  <[domain:]bus:devid.func>" -i dpdk -i dpdk -p http "-p" bstats -p tls -o "ipfix;h=127.0.0.1"`

# Read packets using DPDK input interface as secondary process with shared memory (DPDK rings) - in this case, 4 DPDK rings are used
`./ipfixprobe -i 'dpdk-ring;r=rx_ipfixprobe_0;e= --proc-type=secondary' -i 'dpdk-ring;r=rx_ipfixprobe_1' -i 'dpdk-ring;r=rx_ipfixprobe_2' -i 'dpdk-ring;r=rx_ipfixprobe_3' -o 'text'`

Build

Requirements

To compile DPDK interfaces, make sure you have DPDK libraries (and development files) installed and set the PKG_CONFIG_PATH environment variable if necessary. You can obtain the latest DPDK at http://core.dpdk.org/download/ Use --with-dpdk parameter of the configure script to enable it.

Source codes

This project uses a standard process of:

git clone --recurse-submodules https://github.com/CESNET/ipfixprobe
cd ipfixprobe
autoreconf -i
./configure
make
sudo make install

Check ./configure --help for more details and settings.

RPM packages

RPM package can be created in the following versions using --with parameter of rpmbuild:

These parameters affect required dependencies of the RPM and build process.

The default configuration of the RPM can be created using simply: make rpm

Alternative versions (described in the following section) can be created by:

We use COPR infrastructure to build and serve RPM packages for EPEL9. It is not possible to pass arguments to rpmbuild, so there is an option in configure to enforce NEMEA dependency:

./configure --enable-coprrpm && make srpm

The output source RPM can be uploaded to copr.

To install ipfixprobe with NEMEA dependency from binary RPM packages, it is possible to follow instructions on: https://copr.fedorainfracloud.org/coprs/g/CESNET/NEMEA/

Telemetry

ipfixprobe can expose telemetry data using the appFs library, which leverages the fuse3 library (filesystem in userspace) to allow telemetry data to be accessed and manipulated through standard filesystem operations.

Input / Output of the flow exporter

The availability of the input and output interfaces depends on the ipfixprobe build settings. By default, we provide RPM package with pcap and raw inputs. The default provided outpus are ipfix and text.

When the project is configured with ./configure --with-nemea, the flow exporter supports NEMEA output via TRAP IFC besides the default IPFIX output. For more information about NEMEA, visit https://nemea.liberouter.org.

The flow exporter supports compilation with libpcap (./configure --with-pcap), which allows for receiving packets from PCAP file or network interface card.

When the project is configured with ./configure --with-ndp, it is prepared for high-speed packet transfer from special HW acceleration FPGA cards. For more information about the cards, visit COMBO cards or contact us.

Output

There are several currently available output plugins, such as:

The output flow records are composed of information provided by the enabled plugins (using -p parameter, see Flow Data Extension - Processing Plugins).

See ipfixprobe -h output for more information and complete list of output plugins and their parameters.

LZ4 compression: ipfix plugin supports LZ4 compression algorithm over tcp. See plugin's help for more information.

Possible issues

Flows are not send to output interface when reading small pcap file (NEMEA output)

Turn off message buffering using buffer=off option and set timeout=WAIT on output interfaces.

./ipfixprobe -i 'pcap;file=traffic.pcap' -o 'unirec;i=u:out:timeout=WAIT:buffer=off'