nprobe doesn't start on Centos after update 7.5.170313-5685

[johnydoo]

Nprobe doesn’t start anymore after we updated to version 7.5.170313-5685. Everything was working fine before the update. The log file doesn’t give any clues, so it looks like a sudden crash.

Systemctl status reports active (exited), ps lists the process for a few seconds and after that it’s gone.

CentOS Linux release 7.3.1611 (Core) nProbe 7.5.170313-5685

[root@servntop nprobe]# systemctl status nprobe
nprobe.service - Start/stop nprobe program
Loaded: loaded (/etc/systemd/system/nprobe.service; enabled; vendor preset: disabled)
Active: active (exited) (Result: signal) since Mon 2017-03-13 14:21:34 CET; 21min ago
Main PID: 2588 (code=killed, signal=SEGV)
CGroup: /system.slice/nprobe.service

Mar 13 14:21:34 servntop systemd[1]: Starting Start/stop nprobe program...
Mar 13 14:21:34 servntop nprobe[2537]: Starting nProbe none
Mar 13 14:21:34 servntop systemd[1]: Started Start/stop nprobe program.
Mar 13 14:22:12 servntop systemd[1]: nprobe.service: main process exited, code=killed, status=11/SEGV

Logfile:

13/Mar/2017 14:38:08 [nprobe.c:3553] Valid nProbe license found
13/Mar/2017 14:38:08 [nprobe.c:5363] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ?
13/Mar/2017 14:38:08 [nprobe.c:5366] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ?
13/Mar/2017 14:38:08 [nprobe.c:5465] Welcome to nProbe v.7.5.170313 ($Revision: 5685 $) for x86_64-unknown-linux-gnu with native PF_RING acceleration
13/Mar/2017 14:38:08 [nprobe.c:5475] Running on CentOS Linux release 7.3.1611 (Core)
13/Mar/2017 14:38:08 [nprobe.c:5486] [LICENSE] nProbe SystemId: xxx
13/Mar/2017 14:38:08 [nprobe.c:5600] Sample rate [packet: 1][flow: 1]
13/Mar/2017 14:38:08 [nprobe.c:7885] Welcome to nProbe v.7.5.170313 for x86_64-unknown-linux-gnu
13/Mar/2017 14:38:08 [plugin.c:1068] 0 plugin(s) enabled
13/Mar/2017 14:38:08 [nprobe.c:7412] Non IPv4/v6 traffic is discarded according to the template
13/Mar/2017 14:38:08 [util.c:430] GeoIP: loaded AS config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat
13/Mar/2017 14:38:08 [util.c:441] GeoIP: loaded AS IPv6 config file /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat
13/Mar/2017 14:38:08 [nprobe.c:8061] IPv6 traffic will NOT be exported/accounted by this probe
13/Mar/2017 14:38:08 [nprobe.c:8062] due to configuration options (e.g. use NetFlow v9)
13/Mar/2017 14:38:08 [nprobe.c:8063] Please use -V to set the version to other than NetFlow V5
13/Mar/2017 14:38:08 [nprobe.c:8216] Not capturing packet from interface (collector mode)
13/Mar/2017 14:38:08 [util.c:4096] Initializing ZMQ as server
13/Mar/2017 14:38:08 [util.c:4139] Succesfully created ZMQ endpoint tcp://*:5556
13/Mar/2017 14:38:08 [util.c:3185] nProbe changed user to 'nobody'
13/Mar/2017 14:38:08 [collect.c:143] Flow collector listening on port 9991 (IPv4/v6)
13/Mar/2017 14:38:08 [nprobe.c:8442] nProbe started successfully

[johnydoo]

Update log:

[root@servntop nprobe]# cat /var/log/yum.log | grep probe
Mar 08 08:42:10 Updated: nprobe-7.5.170308-5676.x86_64
Mar 13 14:09:23 Updated: nprobe-7.5.170313-5685.x86_64

7.5.170308-5676 -> 7.5.170313-5685

What changed between these versions and how do we fix this?

[simonemainardi]

please generate a core dump and send us the link. A trace would be useful as well. See http://www.ntop.org/support/faq/how-to-report-a-bug-on-ntop-apps/

[johnydoo]

There is no core file created with ulimit. When started from the command-line it reported this after a few seconds:

13/Mar/2017 15:29:28 [collect.c:1663] WARNING: fieldId 33000 size (12) might be cut to 8 Segmentation fault

[johnydoo]

How to gather more information about the segmentation fault?

[simonemainardi]

please enclose nprobe config file used

[johnydoo]

--zmq="tcp://*:5556" --collector-port=9991 -n=none -i=none -V 9 -T "%IPV4_SRC_ADDR %L4_SRC_PORT %IPV4_DST_ADDR %L4_DST_PORT %PROTOCOL %IN_BYTES %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %IN_PKTS %OUT_PKTS %IP_PROTOCOL_VERSION %APPLICATION_ID %L7_PROTO_NAME %ICMP_TYPE"

[johnydoo]

Same result with:

/usr/local/bin/nprobe --zmq="tcp://*:5556" --collector-port=9991 -i=none

In the previous nProbe version we received this error:

06/Mar/2017 09:02:25 [collect.c:1010] WARNING: Too many template fields (285): skept [pktId: 2245]

(sending device is Cisco ASA firewall using netflow version 9)

Could it be related? Now we only get this

14/Mar/2017 09:27:00 [collect.c:1663] WARNING: fieldId 33000 size (12) might be cut to 8
Segmentation fault

[johnydoo]

Can't upload zip file, so changed the extension to txt.

capture.zip.txt

[lucaderi]

Fixed. Please resync

[johnydoo]

@lucaderi Just updated nProbe to version 7.5.170314-5686 and it's not fixed.