Improve IPS performances

[filippocarletti]

Some IPS performance improvements could be achieved:

1. tuning the configuration
2. better network flow processing on cpu cores

Prior to adjusting the configuration, we should adopt some new default for suricata 4.1 (which we are now using). This should not change the IPS behavior and silence one warning at startup (<Warning> - [ERRCODE: SC_WARN_DEFAULT_WILL_CHANGE(317)] - in 5.0 the default for decoder event stats will go from 'decoder.<proto>.<event>' to 'decoder.event.<proto>.<event>'. See ticket #2225. To suppress this message, set stats.decoder-events-prefix in the yaml.).

Proposed solution

1. new default
2. suricata.yaml tuning
3. pin network flows to cpu

[nethbot]

in 7.8.2003/testing:

• nethserver-suricata-2.1.2-1.2.gdd07a05.ns7.noarch.rpm x86_64 armhfp aarch64

[nethbot]

in 7.8.2003/testing:

• nethserver-suricata-2.1.2-1.5.gb250113.ns7.noarch.rpm x86_64 armhfp aarch64

[nethbot]

in 7.8.2003/testing:

• nethserver-suricata-2.1.2-1.8.gddbc787.ns7.noarch.rpm x86_64 armhfp aarch64

[filippocarletti]

Test cases

The environment to verify the performance enhancements is ideally a high bandwidth connection and a slow cpu system. This env cuts bandwidth, especially with parallel downloads (some speedtests use parallel downloads).

Verify regressions After running suricata for a while check that alerts are logged (in /var/log/suricata/fast.log and evebox).

Some useful commands

• check that only alert and drop event_type are logged: jq . /var/log/suricata/eve.json | grep event_type | sort | uniq -c
• Monitor suricata threads: top -H -p $(cat /run/suricata.pid )
• suricata process using multiple queues (-q 0 -q 1 ... up to numbers of cpu cores) ps -fwww $(cat /run/suricata.pid ) Output with 4 cores: UID PID PPID C STIME TTY STAT TIME CMD suricata 25320 1 16 16:56 ? Ssl 0:06 /sbin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid -q 0 -q 1 -q 2 -q 3 --user suricata
• kernel distributing traffic to all cores

  iptables -vnL loc2fw | grep NFQUEUE
  1122  119K NFQUEUE    all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x10/0x10 NFQUEUE balance 0:3 bypass cpu-fanout`

  Note: balance 0:3 means 4 cores

[gsanchietti]

Tested on a machines with 1, 2 and 4 cores: verified.

Results on a machine with 2 cores

shorewall:

INLINE all+ all+ - - - - - - !0x10/0x10; -j NFQUEUE --queue-bypass --queue-balance 0:1 --queue-cpu-fanout

ps:

suricata 24458     1  7 09:03 ?        Ssl    0:55 /sbin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid -q 0 -q 1 --user suricata

log:

jq . /var/log/suricata/eve.json | grep event_type | sort | uniq -c
    155   "event_type": "alert",
    132   "event_type": "drop",

top:

24511 suricata  20   0  812240 268600   6008 S  0.7 14.5   0:08.48 W-Q1                                                                                                      
24508 suricata  20   0  812240 268600   6008 S  0.3 14.5   0:04.05 W-Q0  

Results on a machine with 1 core

shorewall:

INLINE all+ all+ - - - - - - !0x10/0x10; -j NFQUEUE --queue-bypass 

ps:

suricata  3088     1  1 07:27 ?        Ssl    0:00 /sbin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid -q 0 --user suricata

log:

jq . /var/log/suricata/eve.json | grep event_type | sort | uniq -c
    1   "event_type": "alert",

top:

3091 suricata  20   0  517472  44512   5444 S  0.0  4.4   0:00.00 W-Q0                                                                                                      

[nethbot]

in 7.8.2003/updates:

• nethserver-suricata-2.2.0-1.ns7.noarch.rpm x86_64 armhfp aarch64