search

ntop / ntopng

Web-based Traffic and Security Network Traffic Monitoring
http://www.ntop.org
GNU General Public License v3.0
6.14k stars 647 forks source link

ntopng crashes and restarts every few minutes on signal 7 (bus error) #7887

Closed rreiner closed 10 months ago

rreiner commented 10 months ago

Environment:

What happened: ntopng's main process dies on signal 7 (bus error ).

The installation was previously stable.

Each time the system log shows a sequence like this:

Oct 09 13:05:50 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 09 13:05:50 pinkeye systemd[1]: ntopng.service: Failed with result 'signal'.
Oct 09 13:05:55 pinkeye systemd[1]: ntopng.service: Service RestartSec=5s expired, scheduling restart.
Oct 09 13:05:55 pinkeye systemd[1]: ntopng.service: Scheduled restart job, restart counter is at 379.

This is happening approximately every 5 minutes, even when there is no user interaction with ntopng.

How did you reproduce it? Wait 5 minutes and check the logs:

$ sudo journalctl -n 1000 -u ntopng | grep BUS
Oct 09 13:45:49 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 09 13:50:47 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 09 13:55:48 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 09 14:00:48 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 09 14:05:48 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 09 14:10:49 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 09 14:15:48 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 09 14:20:47 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 09 14:25:48 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 09 14:30:49 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 09 14:35:48 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS

Debug Information: Longer continuous extract from the journal, for context:

-- Logs begin at Sat 2023-10-07 01:50:48 EDT, end at Mon 2023-10-09 13:08:01 EDT. --
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [startup.lua:118] [lists_utils.lua:466] Failure loading host 'lwwfechxdr8aiq0bbhtrxry7i1c8itnz' category '100' in list 'ThreatFox'
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [startup.lua:118] [lists_utils.lua:466] Failure loading host 'ddkkba0zqra9dtqunixbqaa8olgtkc5j' category '100' in list 'ThreatFox'
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [startup.lua:118] [lists_utils.lua:753] Category Lists (23196 hosts, 11777 IPs, 0 JA3) loaded in 2 sec
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [startup.lua:122] Initializing device polices...
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [startup.lua:138] Initializing alerts...
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [startup.lua:147] Initializing timeseries...
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [startup.lua:235] Completed startup.lua
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [PeriodicActivities.cpp:167] Found 10 activities
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'tcp_issues_generic': skipping it
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'tcp_issues_generic': skipping it
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'suspicious_file_transfer': skipping it
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'suspicious_file_transfer': skipping it
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'suspicious_tcp_probing': skipping it
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'suspicious_tcp_probing': skipping it
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'nedge_blocked': skipping it
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'nedge_blocked': skipping it
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'ndpi_ssh_obsolete': skipping it
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'ndpi_ssh_obsolete': skipping it
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'udp_unidirectional': skipping it
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'udp_unidirectional': skipping it
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'suspicious_tcp_syn_probing': skipping it
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'suspicious_tcp_syn_probing': skipping it
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'potentially_dangerous': skipping it
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'potentially_dangerous': skipping it
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [HostChecksLoader.cpp:203] Unable to find host check external_host_script
Oct 09 12:56:09 pinkeye ntopng[20301]: 09/Oct/2023 12:56:09 [NetworkInterface.cpp:3698] Started packet polling on interface eth0 [id: 0]...
Oct 09 13:00:02 pinkeye ntopng[20301]: [LINT] validateParameter failed for parameter [network][function]
Oct 09 13:00:02 pinkeye ntopng[20301]:  string -1
Oct 09 13:00:53 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 09 13:00:54 pinkeye systemd[1]: ntopng.service: Failed with result 'signal'.
Oct 09 13:00:59 pinkeye systemd[1]: ntopng.service: Service RestartSec=5s expired, scheduling restart.
Oct 09 13:00:59 pinkeye systemd[1]: ntopng.service: Scheduled restart job, restart counter is at 378.
Oct 09 13:00:59 pinkeye systemd[1]: Stopped ntopng high-speed web-based traffic monitoring and analysis tool.
Oct 09 13:00:59 pinkeye systemd[1]: Starting ntopng high-speed web-based traffic monitoring and analysis tool...
Oct 09 13:00:59 pinkeye systemd[1]: Started ntopng high-speed web-based traffic monitoring and analysis tool.
Oct 09 13:01:00 pinkeye ntopng[20868]: 09/Oct/2023 13:01:00 [Redis.cpp:164] Successfully connected to redis 127.0.0.1:6379@0
Oct 09 13:01:00 pinkeye ntopng[20868]: 09/Oct/2023 13:01:00 [Redis.cpp:164] Successfully connected to redis 127.0.0.1:6379@0
Oct 09 13:01:00 pinkeye ntopng[20868]: 09/Oct/2023 13:01:00 [NtopPro.cpp:333] [LICENSE] No license file found /etc/ntopng.license: reading license from redis
Oct 09 13:01:00 pinkeye ntopng[20868]: 09/Oct/2023 13:01:00 [NtopPro.cpp:513] [LICENSE] Unable to validate license [Empty license file]
Oct 09 13:01:00 pinkeye ntopng[20868]: 09/Oct/2023 13:01:00 [NtopPro.cpp:583] WARNING: [LICENSE] Invalid license [Empty license file]
Oct 09 13:01:00 pinkeye ntopng[20868]: 09/Oct/2023 13:01:00 [NtopPro.cpp:583] WARNING: [LICENSE] Invalid license [Empty license file]
Oct 09 13:01:00 pinkeye ntopng[20868]: 09/Oct/2023 13:01:00 [NtopPro.cpp:605] WARNING: [LICENSE] ntopng will now run in Enterprise L edition for 10 minutes
Oct 09 13:01:00 pinkeye ntopng[20868]: 09/Oct/2023 13:01:00 [NtopPro.cpp:605] WARNING: [LICENSE] ntopng will now run in Enterprise L edition for 10 minutes
Oct 09 13:01:00 pinkeye ntopng[20868]: 09/Oct/2023 13:01:00 [NtopPro.cpp:610] WARNING: [LICENSE] before returning to community mode
Oct 09 13:01:00 pinkeye ntopng[20868]: 09/Oct/2023 13:01:00 [NtopPro.cpp:610] WARNING: [LICENSE] before returning to community mode
Oct 09 13:01:00 pinkeye ntopng[20868]: 09/Oct/2023 13:01:00 [NtopPro.cpp:612] WARNING: [LICENSE] You can buy a permanent license at http://shop.ntop.org
Oct 09 13:01:00 pinkeye ntopng[20868]: 09/Oct/2023 13:01:00 [NtopPro.cpp:612] WARNING: [LICENSE] You can buy a permanent license at http://shop.ntop.org
Oct 09 13:01:00 pinkeye ntopng[20868]: 09/Oct/2023 13:01:00 [NtopPro.cpp:617] WARNING: [LICENSE] or run ntopng in community mode starting
Oct 09 13:01:00 pinkeye ntopng[20868]: 09/Oct/2023 13:01:00 [NtopPro.cpp:617] WARNING: [LICENSE] or run ntopng in community mode starting
Oct 09 13:01:00 pinkeye ntopng[20868]: 09/Oct/2023 13:01:00 [NtopPro.cpp:619] WARNING: [LICENSE] ntopng --community
Oct 09 13:01:00 pinkeye ntopng[20868]: 09/Oct/2023 13:01:00 [NtopPro.cpp:619] WARNING: [LICENSE] ntopng --community
Oct 09 13:01:00 pinkeye ntopng[20868]: 09/Oct/2023 13:01:00 [Radius.cpp:255] No Radius server configured for authentication or accounting [Auth: 127.0.0.1:1812][Acct: ]
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [PcapInterface.cpp:111] Reading packets from eth0 [id: 0]
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [Ntop.cpp:2762] Registered interface eth0 [id: 0]
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [main.cpp:366] PID stored in file /var/run/ntopng.pid
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [Geolocation.cpp:170] Loaded database dbip-city-lite.mmdb [/usr/share/ntopng/httpdocs/geoip//dbip-city-lite.mmdb][ip_version: 6]
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [Geolocation.cpp:170] Loaded database dbip-asn-lite.mmdb [/usr/share/ntopng/httpdocs/geoip//dbip-asn-lite.mmdb][ip_version: 6]
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [Geolocation.cpp:109] Using geolocation provided by DB-IP (https://db-ip.com)
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [HTTPserver.cpp:1623] Found TLS certificate /usr/share/ntopng/httpdocs/ssl/ntopng-cert.pem
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [HTTPserver.cpp:1914] Web server dirs [/usr/share/ntopng/httpdocs][/usr/share/ntopng/scripts]
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [HTTPserver.cpp:1919] HTTP server listening on 3000
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [Utils.cpp:860] User changed to ntopng
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [NetworkInterface.cpp:3448] Started flow user script hooks loop on interface eth0 [id: 0]...
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [NetworkInterface.cpp:3508] Started host user script hooks loop on interface eth0 [id: 0]...
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [main.cpp:441] Working directory: /ntopngdata
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [main.cpp:443] Scripts/HTML pages directory: /usr/share/ntopng
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [Ntop.cpp:528] Welcome to ntopng armv7l v.5.7.231008 (dev:05a099b6071b7c58bef630b1f3cbabe79a4e9ad6:20231008)
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [Ntop.cpp:537] Built on Raspbian GNU/Linux 10 (buster)
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [Ntop.cpp:539] (C) 1998-23 ntop
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [NtopPro.cpp:820] [LICENSE] System Id:        L130FA343499602D2--OL
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [NtopPro.cpp:822] [LICENSE] Edition:        Community
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [NtopPro.cpp:824] [LICENSE] License Type:        Time-Limited [Empty license file] License
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [NtopPro.cpp:853] [LICENSE] Validity:        Until Mon Oct  9 13:11:00 2023
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [Ntop.cpp:1058] Adding fe80::e65f:1ff:fe0c:9d41/128 as IPv6 interface address for eth0
Oct 09 13:01:02 pinkeye ntopng[20868]: 09/Oct/2023 13:01:02 [Ntop.cpp:1070] Adding fe80::e65f:1ff:fe0c:9d41/64 as IPv6 local network for eth0
Oct 09 13:01:04 pinkeye ntopng[20868]: 09/Oct/2023 13:01:04 [PeriodicActivities.cpp:108] Started periodic activities loop...
Oct 09 13:01:06 pinkeye ntopng[20868]: 09/Oct/2023 13:01:06 [startup.lua:35] Processing startup.lua: please hold on...
Oct 09 13:01:06 pinkeye ntopng[20868]: 09/Oct/2023 13:01:06 [startup.lua:39] [startup.lua:26] Started Top Sites aggregation
Oct 09 13:01:06 pinkeye ntopng[20868]: 09/Oct/2023 13:01:06 [startup.lua:39] [startup.lua:29] Top Sites aggregation done
Oct 09 13:01:07 pinkeye ntopng[20868]: 09/Oct/2023 13:01:07 [startup.lua:118] [lists_utils.lua:831] Refreshing category lists...
Oct 09 13:01:08 pinkeye ntopng[20868]: 09/Oct/2023 13:01:08 [startup.lua:118] [lists_utils.lua:466] Failure loading host 'IP address' category '100' in list 'Stratosphere Lab'
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [startup.lua:118] [lists_utils.lua:466] Failure loading host '56565' category '100' in list 'ThreatFox'
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [startup.lua:118] [lists_utils.lua:466] Failure loading host 'noluyoruzawk' category '100' in list 'ThreatFox'
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [startup.lua:118] [lists_utils.lua:466] Failure loading host '4040' category '100' in list 'ThreatFox'
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [startup.lua:118] [lists_utils.lua:466] Failure loading host 'datacikerim' category '100' in list 'ThreatFox'
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [startup.lua:118] [lists_utils.lua:466] Failure loading host 'nicehash' category '100' in list 'ThreatFox'
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [startup.lua:118] [lists_utils.lua:466] Failure loading host 'mpapwpodllalw' category '100' in list 'ThreatFox'
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [startup.lua:118] [lists_utils.lua:466] Failure loading host 'makelovenotmalware.local' category '100' in list 'ThreatFox'
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [startup.lua:118] [lists_utils.lua:466] Failure loading host 'lwwfechxdr8aiq0bbhtrxry7i1c8itnz' category '100' in list 'ThreatFox'
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [startup.lua:118] [lists_utils.lua:466] Failure loading host 'ddkkba0zqra9dtqunixbqaa8olgtkc5j' category '100' in list 'ThreatFox'
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [startup.lua:118] [lists_utils.lua:753] Category Lists (23196 hosts, 11777 IPs, 0 JA3) loaded in 1 sec
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [startup.lua:122] Initializing device polices...
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [startup.lua:138] Initializing alerts...
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [startup.lua:147] Initializing timeseries...
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [startup.lua:235] Completed startup.lua
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [PeriodicActivities.cpp:167] Found 10 activities
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'tcp_issues_generic': skipping it
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'tcp_issues_generic': skipping it
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'suspicious_file_transfer': skipping it
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'suspicious_file_transfer': skipping it
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'suspicious_tcp_probing': skipping it
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'suspicious_tcp_probing': skipping it
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'nedge_blocked': skipping it
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'nedge_blocked': skipping it
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'ndpi_ssh_obsolete': skipping it
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'ndpi_ssh_obsolete': skipping it
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'udp_unidirectional': skipping it
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'udp_unidirectional': skipping it
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'suspicious_tcp_syn_probing': skipping it
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'suspicious_tcp_syn_probing': skipping it
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'potentially_dangerous': skipping it
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'potentially_dangerous': skipping it
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [HostChecksLoader.cpp:203] Unable to find host check external_host_script
Oct 09 13:01:09 pinkeye ntopng[20868]: 09/Oct/2023 13:01:09 [NetworkInterface.cpp:3698] Started packet polling on interface eth0 [id: 0]...
Oct 09 13:05:50 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 09 13:05:50 pinkeye systemd[1]: ntopng.service: Failed with result 'signal'.
Oct 09 13:05:55 pinkeye systemd[1]: ntopng.service: Service RestartSec=5s expired, scheduling restart.
Oct 09 13:05:55 pinkeye systemd[1]: ntopng.service: Scheduled restart job, restart counter is at 379.
Oct 09 13:05:55 pinkeye systemd[1]: Stopped ntopng high-speed web-based traffic monitoring and analysis tool.
Oct 09 13:05:55 pinkeye systemd[1]: Starting ntopng high-speed web-based traffic monitoring and analysis tool...
Oct 09 13:05:55 pinkeye systemd[1]: Started ntopng high-speed web-based traffic monitoring and analysis tool.
Oct 09 13:05:56 pinkeye ntopng[21425]: 09/Oct/2023 13:05:56 [Redis.cpp:164] Successfully connected to redis 127.0.0.1:6379@0
Oct 09 13:05:56 pinkeye ntopng[21425]: 09/Oct/2023 13:05:56 [Redis.cpp:164] Successfully connected to redis 127.0.0.1:6379@0
Oct 09 13:05:56 pinkeye ntopng[21425]: 09/Oct/2023 13:05:56 [NtopPro.cpp:333] [LICENSE] No license file found /etc/ntopng.license: reading license from redis
Oct 09 13:05:56 pinkeye ntopng[21425]: 09/Oct/2023 13:05:56 [NtopPro.cpp:513] [LICENSE] Unable to validate license [Empty license file]
Oct 09 13:05:56 pinkeye ntopng[21425]: 09/Oct/2023 13:05:56 [NtopPro.cpp:583] WARNING: [LICENSE] Invalid license [Empty license file]
Oct 09 13:05:56 pinkeye ntopng[21425]: 09/Oct/2023 13:05:56 [NtopPro.cpp:583] WARNING: [LICENSE] Invalid license [Empty license file]
Oct 09 13:05:56 pinkeye ntopng[21425]: 09/Oct/2023 13:05:56 [NtopPro.cpp:605] WARNING: [LICENSE] ntopng will now run in Enterprise L edition for 10 minutes
Oct 09 13:05:56 pinkeye ntopng[21425]: 09/Oct/2023 13:05:56 [NtopPro.cpp:605] WARNING: [LICENSE] ntopng will now run in Enterprise L edition for 10 minutes
Oct 09 13:05:56 pinkeye ntopng[21425]: 09/Oct/2023 13:05:56 [NtopPro.cpp:610] WARNING: [LICENSE] before returning to community mode
Oct 09 13:05:56 pinkeye ntopng[21425]: 09/Oct/2023 13:05:56 [NtopPro.cpp:610] WARNING: [LICENSE] before returning to community mode
Oct 09 13:05:56 pinkeye ntopng[21425]: 09/Oct/2023 13:05:56 [NtopPro.cpp:612] WARNING: [LICENSE] You can buy a permanent license at http://shop.ntop.org
Oct 09 13:05:56 pinkeye ntopng[21425]: 09/Oct/2023 13:05:56 [NtopPro.cpp:612] WARNING: [LICENSE] You can buy a permanent license at http://shop.ntop.org
Oct 09 13:05:56 pinkeye ntopng[21425]: 09/Oct/2023 13:05:56 [NtopPro.cpp:617] WARNING: [LICENSE] or run ntopng in community mode starting
Oct 09 13:05:56 pinkeye ntopng[21425]: 09/Oct/2023 13:05:56 [NtopPro.cpp:617] WARNING: [LICENSE] or run ntopng in community mode starting
Oct 09 13:05:56 pinkeye ntopng[21425]: 09/Oct/2023 13:05:56 [NtopPro.cpp:619] WARNING: [LICENSE] ntopng --community
Oct 09 13:05:56 pinkeye ntopng[21425]: 09/Oct/2023 13:05:56 [NtopPro.cpp:619] WARNING: [LICENSE] ntopng --community
Oct 09 13:05:56 pinkeye ntopng[21425]: 09/Oct/2023 13:05:56 [Radius.cpp:255] No Radius server configured for authentication or accounting [Auth: 127.0.0.1:1812][Acct: ]
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [PcapInterface.cpp:111] Reading packets from eth0 [id: 0]
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [Ntop.cpp:2762] Registered interface eth0 [id: 0]
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [main.cpp:366] PID stored in file /var/run/ntopng.pid
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [Geolocation.cpp:170] Loaded database dbip-city-lite.mmdb [/usr/share/ntopng/httpdocs/geoip//dbip-city-lite.mmdb][ip_version: 6]
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [Geolocation.cpp:170] Loaded database dbip-asn-lite.mmdb [/usr/share/ntopng/httpdocs/geoip//dbip-asn-lite.mmdb][ip_version: 6]
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [Geolocation.cpp:109] Using geolocation provided by DB-IP (https://db-ip.com)
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [HTTPserver.cpp:1623] Found TLS certificate /usr/share/ntopng/httpdocs/ssl/ntopng-cert.pem
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [HTTPserver.cpp:1914] Web server dirs [/usr/share/ntopng/httpdocs][/usr/share/ntopng/scripts]
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [HTTPserver.cpp:1919] HTTP server listening on 3000
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [Utils.cpp:860] User changed to ntopng
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [NetworkInterface.cpp:3508] Started host user script hooks loop on interface eth0 [id: 0]...
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [NetworkInterface.cpp:3448] Started flow user script hooks loop on interface eth0 [id: 0]...
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [main.cpp:441] Working directory: /ntopngdata
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [main.cpp:443] Scripts/HTML pages directory: /usr/share/ntopng
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [Ntop.cpp:528] Welcome to ntopng armv7l v.5.7.231008 (dev:05a099b6071b7c58bef630b1f3cbabe79a4e9ad6:20231008)
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [Ntop.cpp:537] Built on Raspbian GNU/Linux 10 (buster)
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [Ntop.cpp:539] (C) 1998-23 ntop
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [NtopPro.cpp:820] [LICENSE] System Id:        L130FA343499602D2--OL
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [NtopPro.cpp:822] [LICENSE] Edition:        Community
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [NtopPro.cpp:824] [LICENSE] License Type:        Time-Limited [Empty license file] License
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [NtopPro.cpp:853] [LICENSE] Validity:        Until Mon Oct  9 13:15:56 2023
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [Ntop.cpp:1058] Adding fe80::e65f:1ff:fe0c:9d41/128 as IPv6 interface address for eth0
Oct 09 13:05:58 pinkeye ntopng[21425]: 09/Oct/2023 13:05:58 [Ntop.cpp:1070] Adding fe80::e65f:1ff:fe0c:9d41/64 as IPv6 local network for eth0
Oct 09 13:06:03 pinkeye ntopng[21425]: 09/Oct/2023 13:06:03 [PeriodicActivities.cpp:108] Started periodic activities loop...
Oct 09 13:06:04 pinkeye ntopng[21425]: 09/Oct/2023 13:06:04 [startup.lua:35] Processing startup.lua: please hold on...
Oct 09 13:06:04 pinkeye ntopng[21425]: 09/Oct/2023 13:06:04 [startup.lua:39] [startup.lua:26] Started Top Sites aggregation
Oct 09 13:06:04 pinkeye ntopng[21425]: 09/Oct/2023 13:06:04 [startup.lua:39] [startup.lua:29] Top Sites aggregation done
Oct 09 13:06:06 pinkeye ntopng[21425]: 09/Oct/2023 13:06:06 [startup.lua:118] [lists_utils.lua:831] Refreshing category lists...
Oct 09 13:06:07 pinkeye ntopng[21425]: 09/Oct/2023 13:06:07 [startup.lua:118] [lists_utils.lua:466] Failure loading host 'IP address' category '100' in list 'Stratosphere Lab'
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [startup.lua:118] [lists_utils.lua:466] Failure loading host '56565' category '100' in list 'ThreatFox'
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [startup.lua:118] [lists_utils.lua:466] Failure loading host 'noluyoruzawk' category '100' in list 'ThreatFox'
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [startup.lua:118] [lists_utils.lua:466] Failure loading host '4040' category '100' in list 'ThreatFox'
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [startup.lua:118] [lists_utils.lua:466] Failure loading host 'datacikerim' category '100' in list 'ThreatFox'
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [startup.lua:118] [lists_utils.lua:466] Failure loading host 'nicehash' category '100' in list 'ThreatFox'
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [startup.lua:118] [lists_utils.lua:466] Failure loading host 'mpapwpodllalw' category '100' in list 'ThreatFox'
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [startup.lua:118] [lists_utils.lua:466] Failure loading host 'makelovenotmalware.local' category '100' in list 'ThreatFox'
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [startup.lua:118] [lists_utils.lua:466] Failure loading host 'lwwfechxdr8aiq0bbhtrxry7i1c8itnz' category '100' in list 'ThreatFox'
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [startup.lua:118] [lists_utils.lua:466] Failure loading host 'ddkkba0zqra9dtqunixbqaa8olgtkc5j' category '100' in list 'ThreatFox'
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [startup.lua:118] [lists_utils.lua:753] Category Lists (23196 hosts, 11777 IPs, 0 JA3) loaded in 2 sec
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [startup.lua:122] Initializing device polices...
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [startup.lua:138] Initializing alerts...
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [startup.lua:147] Initializing timeseries...
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [startup.lua:235] Completed startup.lua
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [PeriodicActivities.cpp:167] Found 10 activities
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'tcp_issues_generic': skipping it
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'tcp_issues_generic': skipping it
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'suspicious_file_transfer': skipping it
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'suspicious_file_transfer': skipping it
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'suspicious_tcp_probing': skipping it
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'suspicious_tcp_probing': skipping it
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'nedge_blocked': skipping it
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'nedge_blocked': skipping it
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'ndpi_ssh_obsolete': skipping it
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'ndpi_ssh_obsolete': skipping it
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'udp_unidirectional': skipping it
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'udp_unidirectional': skipping it
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'suspicious_tcp_syn_probing': skipping it
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'suspicious_tcp_syn_probing': skipping it
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'potentially_dangerous': skipping it
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [FlowChecksLoader.cpp:293] WARNING: Unable to find flow check 'potentially_dangerous': skipping it
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [HostChecksLoader.cpp:203] Unable to find host check external_host_script
Oct 09 13:06:08 pinkeye ntopng[21425]: 09/Oct/2023 13:06:08 [NetworkInterface.cpp:3698] Started packet polling on interface eth0 [id: 0]...
lucaderi commented 10 months ago

Please update ntopng and report

rreiner commented 10 months ago

Updated to v.5.7.231010 No change in the problem.

Oct 10 17:00:48 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 10 17:05:48 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 10 17:10:48 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 10 17:15:48 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 10 17:20:46 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 10 17:25:47 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 10 17:30:49 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 10 17:35:49 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 10 17:40:48 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 10 17:45:49 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
lucaderi commented 10 months ago

@rreiner Can you please download https://www.dropbox.com/scl/fi/gsygutkjt3y0w4rexipy7/ntopng-rpi-arm64-debug.tgz?rlkey=csjut57lau1jhoh0sjhe1m29f&dl=0 and run it (instead of the binary you are using) then paste here the crash report? Thanks

rreiner commented 10 months ago

@lucaderi That's a 64-bit binary; but I'm running a 32-bit version of the OS Sorry if my "architecture" tag misled you in the OP; that's the hardware architecture, but it's running an older install with the 32-bit version of Raspbian

lucaderi commented 10 months ago

can you please update the package from packages.ntop.org first and report if it still crashing (we have fixed a bug yesterday) ? If so I will create a debug for 32 bit

rreiner commented 10 months ago

Now running v.5.7.231012 No change in the crashes -- they continue to occur almost exactly 5 minutes after each restart.

Oct 12 17:35:46 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 12 17:40:46 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 12 17:45:48 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 12 17:50:47 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
Oct 12 17:55:48 pinkeye systemd[1]: ntopng.service: Main process exited, code=killed, status=7/BUS
rreiner commented 10 months ago

Received the update to v5.7.231015.

Crashes continue every 5 minutes.

lucaderi commented 10 months ago

Can you please try to crash https://www.dropbox.com/s/gey08a37qgn294l/ntopng-debug.tgz?dl=0 and report? Thank you

lucaderi commented 10 months ago

@rreiner Do you have an update perhaps?

rreiner commented 10 months ago

Sorry, I'm traveling this weekend, can get to this Sunday evening or Monday

-- Sent from my phone

On Sat, Oct 21, 2023, 11:14 Luca Deri @.***> wrote:

@rreiner https://github.com/rreiner Do you have an update perhaps?

— Reply to this email directly, view it on GitHub https://github.com/ntop/ntopng/issues/7887#issuecomment-1773826208, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC6TFYNKDUJ4YABSJUTQ5WTYAPRFHAVCNFSM6AAAAAA5ZDK46CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZTHAZDMMRQHA . You are receiving this because you were mentioned.Message ID: @.***>

rreiner commented 10 months ago

It's still crashing in the same way with the debug binary.

Is it writing some log data somewhere that I can retrieve and send to you?

lucaderi commented 10 months ago

You should use the debug binary I have sent you to generate a coredump and share it with me for inspection. Is this possible?

rreiner commented 10 months ago

@lucaderi Here is the core file https://www.dropbox.com/scl/fi/1kggiffo7iu7kpmz3zj4j/core-ntopng-S-upload-7-997-993-2336-1698012647.gz?rlkey=qwylfv901grfx4uvn22fjjkm7&dl=0

lucaderi commented 10 months ago

Are you sure this core was generated with the debug binary I have sent you? It says

$ file core-ntopng-S-upload-7-997-993-2336-1698012647 core-ntopng-S-upload-7-997-993-2336-1698012647: ELF 32-bit LSB core file, ARM, version 1 (SYSV), SVR4-style, from '/usr/bin/ntopng /etc/ntopng/ntopng.conf', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/usr/bin/ntopng', platform: 'v7l'

I have sent you ntopng-debug and I expected to see this binary used in the core.

rreiner commented 10 months ago

Yes, it was generated with the debug binary. I renamed it to ntopng. Conf file attached (renamed to ntopng.conf.txt so that Github allows the file type)

ntopng.conf.txt

lucaderi commented 10 months ago

Can you please also attach /var/lib/ntopng/protos.txt ?

rreiner commented 10 months ago

protos.txt

lucaderi commented 10 months ago

@rreiner I haven't managed to crash it nor to understand the culprit with you core dump. Can you please mail me (deri@ntop.org) so that we can arrange an interactive session? Thanks

rreiner commented 10 months ago

@lucaderi Looks like the corrupted rrd file explanation is correct -- after deleting two more rrd files, I am seeing no more crashes. Thanks for the assistance!

lucaderi commented 10 months ago

Thanks for reporting