search

ntop / ntopng

Web-based Traffic and Security Network Traffic Monitoring
http://www.ntop.org
GNU General Public License v3.0
6.24k stars 654 forks source link

Segmentation fault #3432

Closed christianbj87 closed 4 years ago

christianbj87 commented 4 years ago

Hello, We are having problems with the nProbe over the Linux operating system. The error it indicates is: "Segmentation fault"

sudo nprobe --nflite 2055:16 -i none -n 10.1.11.34:9996

17/Feb/2020 10:21:03 [plugin.c:181] No plugins found in ./plugins 17/Feb/2020 10:21:03 [plugin.c:189] Loading 24 plugins [.so] from /usr/local/lib/nprobe/plugins 17/Feb/2020 10:21:03 [plugin.c:884] Unable to enable plugin DHCP Protocol [/etc/nprobe.license.dhcp]: Missing license file 17/Feb/2020 10:21:03 [plugin.c:884] Unable to enable plugin Diameter Protocol [/etc/nprobe.license.diameter]: Missing license file 17/Feb/2020 10:21:03 [plugin.c:884] Unable to enable plugin DNS/LLMNR Protocol [/etc/nprobe.license.dns]: License mismatch error 17/Feb/2020 10:21:03 [plugin.c:884] Unable to enable plugin Export Plugin [/etc/nprobe.license.export]: Missing license file 17/Feb/2020 10:21:03 [plugin.c:884] Unable to enable plugin FTP Protocol [/etc/nprobe.license.ftp]: Missing license file 17/Feb/2020 10:21:03 [plugin.c:884] Unable to enable plugin GTPv0 Signaling Protocol [/etc/nprobe.license.gtpv0]: Missing license file 17/Feb/2020 10:21:03 [plugin.c:884] Unable to enable plugin GTPv1 Signaling Protocol [/etc/nprobe.license.gtpv1]: Missing license file 17/Feb/2020 10:21:03 [plugin.c:884] Unable to enable plugin GTPv2 Signaling Protocol [/etc/nprobe.license.gtpv2]: Missing license file 17/Feb/2020 10:21:03 [plugin.c:884] Unable to enable plugin HTTP Protocol [/etc/nprobe.license.http]: License mismatch error 17/Feb/2020 10:21:03 [plugin.c:884] Unable to enable plugin IMAP Protocol [/etc/nprobe.license.email]: Missing license file 17/Feb/2020 10:21:03 [plugin.c:884] Unable to enable plugin Oracle Protocol [/etc/nprobe.license.oracle]: Missing license file 17/Feb/2020 10:21:03 [plugin.c:884] Unable to enable plugin POP3 Protocol [/etc/nprobe.license.email]: Missing license file 17/Feb/2020 10:21:03 [plugin.c:884] Unable to enable plugin Radius Protocol [/etc/nprobe.license.radius]: Missing license file 17/Feb/2020 10:21:03 [plugin.c:884] Unable to enable plugin RTP Plugin [/etc/nprobe.license.voip]: Missing license file 17/Feb/2020 10:21:03 [plugin.c:884] Unable to enable plugin S1AP Protocol [/etc/nprobe.license.S1AP]: Missing license file 17/Feb/2020 10:21:03 [plugin.c:884] Unable to enable plugin SIP Plugin [/etc/nprobe.license.voip]: Missing license file 17/Feb/2020 10:21:03 [plugin.c:884] Unable to enable plugin SMTP Protocol [/etc/nprobe.license.email]: Missing license file 17/Feb/2020 10:21:03 [nprobe.c:4194] Valid nProbe Pro license found 17/Feb/2020 10:21:03 [nprobe.c:6027] IMPORTANT: Enabling NflitePlugin will also enable IP address forging, thus 17/Feb/2020 10:21:03 [nprobe.c:6028] IMPORTANT: flows appear as they were sent from the NflitePlugin-enabled switch 17/Feb/2020 10:21:03 [nprobe.c:3759] Exporting flows towards 10.1.11.34:9996 using UDP 17/Feb/2020 10:21:03 [nprobe.c:6122] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ? 17/Feb/2020 10:21:03 [nprobe.c:6125] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ? 17/Feb/2020 10:21:03 [nprobe.c:6212] Welcome to nProbe Pro v.8.6.200207 ($Revision: 6343 $) for x86_64-unknown-linux-gnu with native PF_RING acceleration 17/Feb/2020 10:21:03 [nprobe.c:6222] Running on CentOS Linux release 7.7.1908 (Core) 17/Feb/2020 10:21:03 [nprobe.c:6233] [LICENSE] nProbe SystemId: 689ED9C776066B16 17/Feb/2020 10:21:03 [nprobe.c:6300] Sample rate [packet: 1][flow collection/export: 1/1] 17/Feb/2020 10:21:03 [modbusPlugin.c:102] [MODBUS] Idle flow timeout set to 120 sec 17/Feb/2020 10:21:03 [nflitePlugin.c:879] ERROR: Flow collector port 2055/IPv6 already in use ? [Address family not supported by protocol/97]: disabling collection over IPv6 17/Feb/2020 10:21:03 [nflitePlugin.c:903] [NFLite] Listening on port range 2055-2070 (16) 17/Feb/2020 10:21:03 [nprobe.c:9013] Welcome to nProbe v.8.6.200207 for x86_64-unknown-linux-gnu 17/Feb/2020 10:21:03 [nprobe.c:8001] You selected v9/IPFIX without specifying a template (-T). 17/Feb/2020 10:21:03 [nprobe.c:8002] The default template will be used 17/Feb/2020 10:21:03 [nprobe.c:8009] Using NetFlow Packet Payload Len: 1472 17/Feb/2020 10:21:03 [plugin.c:1278] 1 plugin(s) enabled 17/Feb/2020 10:21:03 [nprobe.c:8461] Each flow is 71 bytes long 17/Feb/2020 10:21:03 [nprobe.c:8462] The # flows per packet has been set to 19 17/Feb/2020 10:21:03 [nprobe.c:8465] IP TOS is accounted 17/Feb/2020 10:21:03 [nprobe.c:8491] Non IPv4/v6 traffic is discarded according to the template 17/Feb/2020 10:21:03 [nprobe.c:9274] Flows ASs will not be computed (no GeoDB files loaded) 17/Feb/2020 10:21:03 [nprobe.c:9379] Not capturing packet from interface (collector mode) 17/Feb/2020 10:21:03 [plugin.c:960] Enabling plugin Netflow-Lite Plugin 17/Feb/2020 10:21:03 [nprobe.c:9629] nProbe started successfully Segmentation fault

No problem when i set one port (2055): sudo nprobe --nflite 2055 -i none -n 10.1.11.34:9996

nprobe -v

Welcome to nProbe v.8.6.200207 (r6343) for x86_64-unknown-linux-gnu with native PF_RING acceleration. Copyright 2002-18 ntop.org

Build OS: CentOS Linux release 7.7.1908 (Core) SystemID: 689ED9C776066B16 GIT rev: 8.6-stable:67c7296f177602e07b6d109f4200f7d2453d8b49:20200207 Edition: nProbe Pro License: gvXHfJ54HsRy6UrWqWILiviLvlqJuwKtBNleQzqvoNrAGgcK6IDYFB+3XTPby+mT5dZRX hyBe+LId+GOOzf4NxWZC3he+ClI5PPC8ZxaAVySE9Ny1c3egE5aoszyk7Mu/66CY23fQY 0McdwpIzaEck+OPOHhw4E/p0624N0xvyddFzOuPO1vSQsQX1oookkm22OGLTFHutvQLSl RDfhtWKAuvKC8Qbaa6HQPAYTWiAgsvTaUom2Cuonzsg4cwPjZadU1v2e5seBGe8VaxnxG iA53tt3GHjrsNPqXw2YY/vxElZIFrQTpLL1/lF+wG0W4Ii2XCBIiAzWBExswevqPQA== License Hash: A4F5F27E636BCCA2F1A93E2FC91F0450161126278950C5729A [valid license] License Type: Permanent License Maintenance: Until Thu Jan 21 17:59:49 2021 [339 days left]

Thanks.

lucaderi commented 4 years ago

@christianbj87 Can you please use the dev version and if not working use the enclosed version and share the core? nprobes-8.7.200217.gz

christianbj87 commented 4 years ago

I already installed dev version, how i run it?

simonemainardi commented 4 years ago

dev versions (and installation instructions) are available at https://packages.ntop.org/centos/

Install the dev and run it as if it was the stable version

christianbj87 commented 4 years ago

I tryed dev versions, i copy the output:

sudo nprobe -v

Welcome to nProbe v.8.7.200218 (r6753) for x86_64-unknown-linux-gnu with native PF_RING acceleration. Copyright 2002-19 ntop.org

Build OS: CentOS Linux release 7.7.1908 (Core) SystemID: 689ED9C776066B16 GIT rev: dev:c25dd94c76dafc3e9a15ed65e36dce9ddf1dc815:20200218 Edition: nProbe Pro

sudo nprobe --nflite 2055:16 -i none -n 10.1.11.34:9996 18/Feb/2020 09:31:51 [plugin.c:177] No plugins found in ./plugins 18/Feb/2020 09:31:51 [plugin.c:185] Loading 23 plugins [.so] from /usr/local/lib/nprobe/plugins 18/Feb/2020 09:31:51 [plugin.c:873] Unable to enable plugin DHCP Protocol [/etc/nprobe.license.dhcp]: Missing license file 18/Feb/2020 09:31:51 [plugin.c:873] Unable to enable plugin Diameter Protocol [/etc/nprobe.license.diameter]: Missing license file 18/Feb/2020 09:31:51 [plugin.c:873] Unable to enable plugin DNS/LLMNR Protocol [/etc/nprobe.license]: Missing license file 18/Feb/2020 09:31:51 [plugin.c:873] Unable to enable plugin Export Plugin [/etc/nprobe.license.export]: Missing license file 18/Feb/2020 09:31:51 [plugin.c:873] Unable to enable plugin FTP Protocol [/etc/nprobe.license.ftp]: Missing license file 18/Feb/2020 09:31:51 [plugin.c:873] Unable to enable plugin GTPv0 Signaling Protocol [/etc/nprobe.license.gtpv0]: Missing license file 18/Feb/2020 09:31:51 [plugin.c:873] Unable to enable plugin GTPv1 Signaling Protocol [/etc/nprobe.license.gtpv1]: Missing license file 18/Feb/2020 09:31:51 [plugin.c:873] Unable to enable plugin GTPv2 Signaling Protocol [/etc/nprobe.license.gtpv2]: Missing license file 18/Feb/2020 09:31:51 [plugin.c:873] Unable to enable plugin HTTP Protocol [/etc/nprobe.license]: Missing license file 18/Feb/2020 09:31:51 [plugin.c:873] Unable to enable plugin IMAP Protocol [/etc/nprobe.license.email]: Missing license file 18/Feb/2020 09:31:51 [plugin.c:873] Unable to enable plugin POP3 Protocol [/etc/nprobe.license.email]: Missing license file 18/Feb/2020 09:31:51 [plugin.c:873] Unable to enable plugin Radius Protocol [/etc/nprobe.license.radius]: Missing license file 18/Feb/2020 09:31:51 [plugin.c:873] Unable to enable plugin RTP Plugin [/etc/nprobe.license.voip]: Missing license file 18/Feb/2020 09:31:51 [plugin.c:873] Unable to enable plugin SIP Plugin [/etc/nprobe.license.voip]: Missing license file 18/Feb/2020 09:31:51 [plugin.c:873] Unable to enable plugin SMTP Protocol [/etc/nprobe.license.email]: Missing license file 18/Feb/2020 09:31:51 [nprobe.c:4530] Valid nProbe Pro license found 18/Feb/2020 09:31:51 [nprobe.c:6478] IMPORTANT: Enabling NflitePlugin will also enable IP address forging, thus 18/Feb/2020 09:31:51 [nprobe.c:6479] IMPORTANT: flows appear as they were sent from the NflitePlugin-enabled switch 18/Feb/2020 09:31:51 [nprobe.c:4020] Exporting flows towards 10.1.11.34:9996 using UDP 18/Feb/2020 09:31:51 [nprobe.c:6578] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ? 18/Feb/2020 09:31:51 [nprobe.c:6581] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ? 18/Feb/2020 09:31:51 [nprobe.c:6668] Welcome to Pro nProbe v.8.7.200218 ($Revision: 6753 $) for x86_64-unknown-linux-gnu with native PF_RING acceleration 18/Feb/2020 09:31:51 [nprobe.c:6679] Running on CentOS Linux release 7.7.1908 (Core) 18/Feb/2020 09:31:51 [nprobe.c:6690] [LICENSE] nProbe SystemId: 689ED9C776066B16 18/Feb/2020 09:31:51 [nprobe.c:6761] Sample rate [packet: 1][flow collection/export: 1/1] 18/Feb/2020 09:31:51 [modbusPlugin.c:104] [MODBUS] Idle flow timeout set to 120 sec 18/Feb/2020 09:31:51 [nflitePlugin.c:879] ERROR: Flow collector port 2055/IPv6 already in use ? [Address family not supported by protocol/97]: disabling collection over IPv6 18/Feb/2020 09:31:51 [nflitePlugin.c:903] [NFLite] Listening on port range 2055-2070 (16) 18/Feb/2020 09:31:51 [nprobe.c:9589] Welcome to nProbe v.8.7.200218 for x86_64-unknown-linux-gnu 18/Feb/2020 09:31:51 [nprobe.c:8546] You selected v9/IPFIX without specifying a template (-T). 18/Feb/2020 09:31:51 [nprobe.c:8547] The default template will be used 18/Feb/2020 09:31:51 [nprobe.c:8554] Using NetFlow Packet Payload Len: 1472 18/Feb/2020 09:31:51 [nprobe.c:8589] Flow export type: unidirectional flows 18/Feb/2020 09:31:51 [plugin.c:1309] 1 plugin(s) enabled 18/Feb/2020 09:31:51 [nprobe.c:9024] Each flow is 71 bytes long 18/Feb/2020 09:31:51 [nprobe.c:9025] The # flows per packet has been set to 19 18/Feb/2020 09:31:51 [nprobe.c:9028] IP TOS is ignored 18/Feb/2020 09:31:51 [nprobe.c:9056] Non IPv4/v6 traffic is discarded according to the template 18/Feb/2020 09:31:51 [nprobe.c:9872] Flows ASs will not be computed (no GeoDB files loaded) 18/Feb/2020 09:31:51 [nprobe.c:9977] Not capturing packet from interface (collector mode) 18/Feb/2020 09:31:51 [plugin.c:961] Enabling plugin Netflow-Lite Plugin 18/Feb/2020 09:31:51 [export.c:543] Using TLV as serialization format 18/Feb/2020 09:31:51 [nprobe.c:10239] nProbe started successfully 18/Feb/2020 09:36:04 [engine.c:1785] WARNING: Internal error: NULL head for index 66329 [num_runs: 1][thread_id: 0] Segmentation fault

emanuele-f commented 4 years ago

Can you get a gdb stack trace as follows?

  1. Install gdb (e.g. sudo yum install gdb)
  2. Start gdb: sudo gdb --args nprobe --nflite 2055:16 -i none -n 10.1.11.34:9996
  3. Execute run to start debunning nprobe
  4. Wait for the crash to occur
  5. Now run bt into gdb to get a stack trace of the crash
christianbj87 commented 4 years ago

Hi Emanuele, i pass the logs for the gdb:

gdb --args nprobe --nflite 2055:16 -i none -n 10.1.11.34:9996 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/local/bin/nprobe...(no debugging symbols found)...done. (gdb) run Starting program: /usr/local/bin/nprobe --nflite 2055:16 -i none -n 10.1.11.34:9996 [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". 18/Feb/2020 12:39:58 [plugin.c:177] No plugins found in ./plugins 18/Feb/2020 12:39:58 [plugin.c:185] Loading 23 plugins [.so] from /usr/local/lib/nprobe/plugins 18/Feb/2020 12:39:58 [plugin.c:873] Unable to enable plugin DHCP Protocol [/etc/nprobe.license.dhcp]: Missing license file 18/Feb/2020 12:39:58 [plugin.c:873] Unable to enable plugin Diameter Protocol [/etc/nprobe.license.diameter]: Missing license file 18/Feb/2020 12:39:58 [plugin.c:873] Unable to enable plugin DNS/LLMNR Protocol [/etc/nprobe.license]: Missing license file 18/Feb/2020 12:39:58 [plugin.c:873] Unable to enable plugin Export Plugin [/etc/nprobe.license.export]: Missing license file 18/Feb/2020 12:39:58 [plugin.c:873] Unable to enable plugin FTP Protocol [/etc/nprobe.license.ftp]: Missing license file 18/Feb/2020 12:39:58 [plugin.c:873] Unable to enable plugin GTPv0 Signaling Protocol [/etc/nprobe.license.gtpv0]: Missing license file 18/Feb/2020 12:39:58 [plugin.c:873] Unable to enable plugin GTPv1 Signaling Protocol [/etc/nprobe.license.gtpv1]: Missing license file 18/Feb/2020 12:39:58 [plugin.c:873] Unable to enable plugin GTPv2 Signaling Protocol [/etc/nprobe.license.gtpv2]: Missing license file 18/Feb/2020 12:39:59 [plugin.c:873] Unable to enable plugin HTTP Protocol [/etc/nprobe.license]: Missing license file 18/Feb/2020 12:39:59 [plugin.c:873] Unable to enable plugin IMAP Protocol [/etc/nprobe.license.email]: Missing license file 18/Feb/2020 12:39:59 [plugin.c:873] Unable to enable plugin POP3 Protocol [/etc/nprobe.license.email]: Missing license file 18/Feb/2020 12:39:59 [plugin.c:873] Unable to enable plugin Radius Protocol [/etc/nprobe.license.radius]: Missing license file 18/Feb/2020 12:39:59 [plugin.c:873] Unable to enable plugin RTP Plugin [/etc/nprobe.license.voip]: Missing license file 18/Feb/2020 12:39:59 [plugin.c:873] Unable to enable plugin SIP Plugin [/etc/nprobe.license.voip]: Missing license file 18/Feb/2020 12:39:59 [plugin.c:873] Unable to enable plugin SMTP Protocol [/etc/nprobe.license.email]: Missing license file 18/Feb/2020 12:39:59 [nprobe.c:4530] Valid nProbe Pro license found 18/Feb/2020 12:39:59 [nprobe.c:6478] IMPORTANT: Enabling NflitePlugin will also enable IP address forging, thus 18/Feb/2020 12:39:59 [nprobe.c:6479] IMPORTANT: flows appear as they were sent from the NflitePlugin-enabled switch 18/Feb/2020 12:39:59 [nprobe.c:4020] Exporting flows towards 10.1.11.34:9996 using UDP 18/Feb/2020 12:39:59 [nprobe.c:6578] WARNING: The output interfaceId is set to 0: did you forget to use -Q perhaps ? 18/Feb/2020 12:39:59 [nprobe.c:6581] WARNING: The input interfaceId is set to 0: did you forget to use -u perhaps ? 18/Feb/2020 12:39:59 [nprobe.c:6668] Welcome to Pro nProbe v.8.7.200218 ($Revision: 6753 $) for x86_64-unknown-linux-gnu with native PF_RING acceleration 18/Feb/2020 12:39:59 [nprobe.c:6679] Running on CentOS Linux release 7.7.1908 (Core) 18/Feb/2020 12:39:59 [nprobe.c:6690] [LICENSE] nProbe SystemId: 689ED9C776066B16 18/Feb/2020 12:39:59 [nprobe.c:6761] Sample rate [packet: 1][flow collection/export: 1/1] 18/Feb/2020 12:39:59 [modbusPlugin.c:104] [MODBUS] Idle flow timeout set to 120 sec 18/Feb/2020 12:39:59 [nflitePlugin.c:879] ERROR: Flow collector port 2055/IPv6 already in use ? [Address family not supported by protocol/97]: disabling collection over IPv6 [New Thread 0x7fffec65c700 (LWP 35757)] [New Thread 0x7fffebe5b700 (LWP 35758)] [New Thread 0x7fffeb65a700 (LWP 35759)] [New Thread 0x7fffeae59700 (LWP 35760)] [New Thread 0x7fffea658700 (LWP 35761)] [New Thread 0x7fffe9e57700 (LWP 35762)] [New Thread 0x7fffe9656700 (LWP 35763)] [New Thread 0x7fffe8e55700 (LWP 35764)] [New Thread 0x7fffe8654700 (LWP 35765)] [New Thread 0x7fffe7e53700 (LWP 35766)] [New Thread 0x7fffe7652700 (LWP 35767)] [New Thread 0x7fffe6e51700 (LWP 35768)] [New Thread 0x7fffe6650700 (LWP 35769)] [New Thread 0x7fffe5e4f700 (LWP 35770)] [New Thread 0x7fffe564e700 (LWP 35771)] [New Thread 0x7fffe4e4d700 (LWP 35772)] 18/Feb/2020 12:39:59 [nflitePlugin.c:903] [NFLite] Listening on port range 2055-2070 (16) 18/Feb/2020 12:39:59 [nprobe.c:9589] Welcome to nProbe v.8.7.200218 for x86_64-unknown-linux-gnu 18/Feb/2020 12:39:59 [nprobe.c:8546] You selected v9/IPFIX without specifying a template (-T). 18/Feb/2020 12:39:59 [nprobe.c:8547] The default template will be used 18/Feb/2020 12:39:59 [nprobe.c:8554] Using NetFlow Packet Payload Len: 1472 18/Feb/2020 12:39:59 [nprobe.c:8589] Flow export type: unidirectional flows 18/Feb/2020 12:39:59 [plugin.c:1309] 1 plugin(s) enabled 18/Feb/2020 12:39:59 [nprobe.c:9024] Each flow is 71 bytes long 18/Feb/2020 12:39:59 [nprobe.c:9025] The # flows per packet has been set to 19 18/Feb/2020 12:39:59 [nprobe.c:9028] IP TOS is ignored 18/Feb/2020 12:39:59 [nprobe.c:9056] Non IPv4/v6 traffic is discarded according to the template 18/Feb/2020 12:39:59 [nprobe.c:9872] Flows ASs will not be computed (no GeoDB files loaded) 18/Feb/2020 12:39:59 [nprobe.c:9977] Not capturing packet from interface (collector mode) 18/Feb/2020 12:39:59 [plugin.c:961] Enabling plugin Netflow-Lite Plugin [New Thread 0x7fffe464c700 (LWP 35774)] [New Thread 0x7fffe3e4b700 (LWP 35775)] 18/Feb/2020 12:39:59 [export.c:543] Using TLV as serialization format 18/Feb/2020 12:39:59 [nprobe.c:10239] nProbe started successfully

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fffe9e57700 (LWP 35762)] 0x00007ffff79262b4 in walkHashList () from /usr/local/lib/libnprobe-8.7.200218.so Missing separate debuginfos, use: debuginfo-install nprobe-8.7.200218-6753.x86_64 (gdb) bt

0 0x00007ffff79262b4 in walkHashList () from /usr/local/lib/libnprobe-8.7.200218.so

1 0x00007ffff7930590 in idleThreadTask () from /usr/local/lib/libnprobe-8.7.200218.so

2 0x00007ffff7929aea in processFlowPacket () from /usr/local/lib/libnprobe-8.7.200218.so

3 0x000000000041989f in ?? ()

4 0x000000000041a1cd in decodePacket ()

5 0x00007fffed7d6c13 in readNFlitePackets () from /usr/local/lib/nprobe/plugins/libnflitePlugin-8.7.200218.so

6 0x00007ffff646ae65 in start_thread () from /lib64/libpthread.so.0

7 0x00007ffff489188d in clone () from /lib64/libc.so.6

(gdb)

christianbj87 commented 4 years ago

Any suggestions?

cardigliano commented 4 years ago

@christianbj87 I moved this to https://github.com/ntop/nProbe/issues/401