search

ntop / ntopng

Web-based Traffic and Security Network Traffic Monitoring
http://www.ntop.org
GNU General Public License v3.0
6.28k stars 656 forks source link

Suddenly ntopng doesn't send any data to InfluxDB #2643

Closed komarEX closed 5 years ago

komarEX commented 5 years ago

Hi Suddenly tonight out ntopng server was getting a lot of data in directory /var/log/ntopng. It seems like due to unknown to me reasons ntopng decided that it won't push any more data into InfluxDB even if everything was working just fine before (no one did anything on the server).

I thought that maybe it's some kind of bug and we just need to upgrade so I did but nothing changed.

Ntopng version: 3.8.190606-6817 InfluxDB version: 1.7.6-1

Our timeseries configuration is as follows: image

Runtime Status: image

If I understand that export system correctly it first dumps data into (in our case) /var/log/ntopng/X(-interface)/ts_export and after that it will push this data into InfluxDB. I've enabled verbose logging on Influx to check if there are some kind of POSTs but there is none. If I reenable configuration it will create database, it will read data from it but it won't push any data into it (not sure why). I've enabled more verbose logging into ntopng but I fail to find anything related to any information regarding why it doesn't save. I see that it runs functions for append data into influx but it doesn't do anything (no errors, no warnings, nothing).

I can provide you with anything you want but first tell me how can I debug it. I've already tried to use tcpdump, strace, verbose logging, server update, reboot, creating new database in influxdb. Nothing helps and I'm out of ideas why it suddenly happened. We already have over 200GB of data in Influx and everything was working fine since today.

emanuele-f commented 5 years ago

Can you see any export failed alerts into ntopng?

komarEX commented 5 years ago

Yes but I've already fixed that and still no luck. After this fix no new alerts regarding influxdb.

There was an error while sending timeseries data to 'http://localhost:8086': partial write: max-values-per-tag limit exceeded (100002/100000): measurement='host:traffic' tag='host' value='XXXXXXX' dropped=1105. This can be fixed by setting 'max-values-per-tag = 0' in the InfluxDB server config.

emanuele-f commented 5 years ago

Can you try to manually execute this commands and post the output here?

curl -i -XPOST "http://localhost:8086/write?db=ntopng" --data-binary "test_write,tag1=val1 value=10 1559828756000000000"

influx -database ntopng -execute 'select * from test_write'
komarEX commented 5 years ago 
root@ntopng:~# curl -i -XPOST "http://localhost:8086/write?db=ntopng" --data-binary "test_write,tag1=val1 value=10 1559828756000000000"
HTTP/1.1 204 No Content
Content-Type: application/json
Request-Id: 9e528b46-8863-11e9-82e3-0015176fac30
X-Influxdb-Build: OSS
X-Influxdb-Version: 1.7.6
X-Request-Id: 9e528b46-8863-11e9-82e3-0015176fac30
Date: Thu, 06 Jun 2019 14:01:45 GMT

root@ntopng:~# influx -database ntopng -execute 'select * from test_write'
name: test_write
time                tag1 value
----                ---- -----
1559828756000000000 val1 10
root@ntopng:~#
emanuele-f commented 5 years ago

Can you set the ntopng log back to "normal", reboot ntopng, and then run sudo journalctl -u ntopng and post here the last log trace?

komarEX commented 5 years ago

Sure.

cze 07 08:58:33 hostersi-ntopng systemd[1]: Starting ntopng high-speed web-based traffic monitoring and analysis tool...
cze 07 08:58:33 hostersi-ntopng systemd[1]: Started ntopng high-speed web-based traffic monitoring and analysis tool.
cze 07 08:58:33 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:33 [Prefs.cpp:1166] Using ElasticSearch for data dump [ntopng][ntopng-%Y.%m.%d][http://localhost:9200/_bulk]
cze 07 08:58:33 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:33 [Ntop.cpp:1902] Setting local networks to (CUT)
cze 07 08:58:33 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:33 [Redis.cpp:127] Successfully connected to redis 127.0.0.1:6379@0
cze 07 08:58:33 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:33 [Redis.cpp:127] Successfully connected to redis 127.0.0.1:6379@0
cze 07 08:58:33 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:33 [NtopPro.cpp:314] [LICENSE] Reading license from /etc/ntopng.license
cze 07 08:58:33 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:33 [NtopPro.cpp:205] [LICENSE] /etc/ntopng.license: unable to validate Enterprise license [License mismatch error]
cze 07 08:58:33 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:33 [NtopPro.cpp:208] [LICENSE] /etc/ntopng.license: found valid Professional license
cze 07 08:58:34 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:34 [PF_RINGInterface.cpp:53] Reading packets from PF_RING v.7.4.0 interface enp2s0f0...
cze 07 08:58:34 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:34 [Ntop.cpp:1994] Registered interface enp2s0f0 [id: 2]
cze 07 08:58:35 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:35 [PF_RINGInterface.cpp:53] Reading packets from PF_RING v.7.4.0 interface enp2s0f1...
cze 07 08:58:35 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:35 [Ntop.cpp:1994] Registered interface enp2s0f1 [id: 3]
cze 07 08:58:35 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:35 [Ntop.cpp:1994] Registered interface view:enp2s0f0,enp2s0f1 [id: 4]
cze 07 08:58:35 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:35 [main.cpp:302] PID stored in file /var/run/ntopng.pid
cze 07 08:58:36 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:36 [HTTPserver.cpp:1179] HTTP logs will be stored on /var/log/ntopng/ntopng_access.log
cze 07 08:58:36 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:36 [Utils.cpp:592] User changed to ntopng
cze 07 08:58:36 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:36 [HTTPserver.cpp:1199] Web server dirs [/usr/share/ntopng/httpdocs][/usr/share/ntopng/scripts]
cze 07 08:58:36 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:36 [HTTPserver.cpp:1208] HTTPS server listening on 443
cze 07 08:58:36 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:36 [main.cpp:396] Logging onto /var/log/ntopng/ntopng.log
cze 07 08:58:36 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:36 [main.cpp:399] Working directory: /var/log/ntopng
cze 07 08:58:36 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:36 [main.cpp:401] Scripts/HTML pages directory: /usr/share/ntopng
cze 07 08:58:36 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:36 [Ntop.cpp:403] Welcome to ntopng x86_64 v.3.8.190606 - (C) 1998-18 ntop.org
cze 07 08:58:36 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:36 [Ntop.cpp:413] Built on Ubuntu 18.04.2 LTS
cze 07 08:58:36 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:36 [NtopPro.cpp:638] [LICENSE] System Id:        8C1263CCB206AB13
cze 07 08:58:36 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:36 [NtopPro.cpp:639] [LICENSE] Edition:        Professional
cze 07 08:58:36 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:36 [NtopPro.cpp:640] [LICENSE] License Type:        Permanent License
cze 07 08:58:36 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:36 [NtopPro.cpp:654] [LICENSE] License Hash:        F88ADEF743DB87DCA088A263F2F71C141575967298FFB63A35
cze 07 08:58:36 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:36 [NtopPro.cpp:664] [LICENSE] Maintenance:        Until Tue Dec 10 09:41:38 2019 [186 days left]
cze 07 08:58:36 hostersi-ntopng ntopng[29540]: 07/Jun/2019 08:58:36 [PeriodicActivities.cpp:72] Started periodic activities loop...
cze 07 09:00:55 hostersi-ntopng ntopng[29540]: 07/Jun/2019 09:00:55 [PeriodicActivities.cpp:113] Each periodic activity script will use 3 threads
cze 07 09:00:55 hostersi-ntopng ntopng[29540]: 07/Jun/2019 09:00:55 [NetworkInterface.cpp:2583] Started packet polling on interface enp2s0f0 [id: 2]...
cze 07 09:00:55 hostersi-ntopng ntopng[29540]: 07/Jun/2019 09:00:55 [NetworkInterface.cpp:2583] Started packet polling on interface enp2s0f1 [id: 3]...
cze 07 09:01:03 hostersi-ntopng ntopng[29540]: 07/Jun/2019 09:01:03 [LuaEngine.cpp:1713] ERROR: Interface didn't reload custom categories on time [iface: view:enp2s0f0,enp2s0f1]
cze 07 09:03:00 hostersi-ntopng ntopng[29540]: 07/Jun/2019 09:01:50 [minute.lua:32] [rrd_5min_dump_utils.lua:279] ERROR: [view:enp2s0f0,enp2s0f1 ]Cannot complete local hosts RRD dump. Disk slow or too many local hosts?
cze 07 09:03:00 hostersi-ntopng ntopng[29540]: 07/Jun/2019 09:03:00 [LuaEngine.cpp:7712] [view:enp2s0f0,enp2s0f1] ERROR: Cannot complete top talkers generation in 1 minute. Is there a huge number of hosts in the system?
cze 07 09:04:00 hostersi-ntopng ntopng[29540]: 07/Jun/2019 09:03:00 [minute.lua:32] [rrd_5min_dump_utils.lua:279] ERROR: [view:enp2s0f0,enp2s0f1 ]Cannot complete local hosts RRD dump. Disk slow or too many local hosts?
cze 07 09:04:00 hostersi-ntopng ntopng[29540]: 07/Jun/2019 09:04:00 [LuaEngine.cpp:7712] [view:enp2s0f0,enp2s0f1] ERROR: Cannot complete top talkers generation in 1 minute. Is there a huge number of hosts in the system?
cze 07 09:05:00 hostersi-ntopng ntopng[29540]: 07/Jun/2019 09:04:00 [minute.lua:32] [rrd_5min_dump_utils.lua:279] ERROR: [view:enp2s0f0,enp2s0f1 ]Cannot complete local hosts RRD dump. Disk slow or too many local hosts?
cze 07 09:05:00 hostersi-ntopng ntopng[29540]: 07/Jun/2019 09:04:13 [minute.lua:32] [rrd_5min_dump_utils.lua:279] ERROR: [enp2s0f1 ]Cannot complete local hosts RRD dump. Disk slow or too many local hosts?
cze 07 09:05:00 hostersi-ntopng ntopng[29540]: 07/Jun/2019 09:05:00 [LuaEngine.cpp:7712] [view:enp2s0f0,enp2s0f1] ERROR: Cannot complete top talkers generation in 1 minute. Is there a huge number of hosts in the system?
cze 07 09:06:00 hostersi-ntopng ntopng[29540]: 07/Jun/2019 09:05:00 [minute.lua:32] [rrd_5min_dump_utils.lua:279] ERROR: [view:enp2s0f0,enp2s0f1 ]Cannot complete local hosts RRD dump. Disk slow or too many local hosts?
cze 07 09:06:00 hostersi-ntopng ntopng[29540]: 07/Jun/2019 09:06:00 [LuaEngine.cpp:7712] [view:enp2s0f0,enp2s0f1] ERROR: Cannot complete top talkers generation in 1 minute. Is there a huge number of hosts in the system?
cze 07 09:07:00 hostersi-ntopng ntopng[29540]: 07/Jun/2019 09:06:00 [minute.lua:32] [rrd_5min_dump_utils.lua:279] ERROR: [view:enp2s0f0,enp2s0f1 ]Cannot complete local hosts RRD dump. Disk slow or too many local hosts?
cze 07 09:07:00 hostersi-ntopng ntopng[29540]: 07/Jun/2019 09:07:00 [LuaEngine.cpp:7712] [view:enp2s0f0,enp2s0f1] ERROR: Cannot complete top talkers generation in 1 minute. Is there a huge number of hosts in the system?
cze 07 09:08:00 hostersi-ntopng ntopng[29540]: 07/Jun/2019 09:07:00 [minute.lua:32] [rrd_5min_dump_utils.lua:279] ERROR: [view:enp2s0f0,enp2s0f1 ]Cannot complete local hosts RRD dump. Disk slow or too many local hosts?
cze 07 09:08:00 hostersi-ntopng ntopng[29540]: 07/Jun/2019 09:07:49 [minute.lua:32] [rrd_5min_dump_utils.lua:279] ERROR: [enp2s0f1 ]Cannot complete local hosts RRD dump. Disk slow or too many local hosts?
cze 07 09:08:00 hostersi-ntopng ntopng[29540]: 07/Jun/2019 09:08:00 [LuaEngine.cpp:7712] [view:enp2s0f0,enp2s0f1] ERROR: Cannot complete top talkers generation in 1 minute. Is there a huge number of hosts in the system?
cze 07 09:09:00 hostersi-ntopng ntopng[29540]: 07/Jun/2019 09:08:00 [minute.lua:32] [rrd_5min_dump_utils.lua:279] ERROR: [view:enp2s0f0,enp2s0f1 ]Cannot complete local hosts RRD dump. Disk slow or too many local hosts?

These errors in the end are there since forever so I don't think that it's a problem. Especially when I have 2 interfaces and 1 view and all 3 of them waste resources in ../ts_export directories.

emanuele-f commented 5 years ago

Please note that you can selectively disable timeseries from the interface settings page https://www.ntop.org/guides/ntopng/web_gui/interfaces.html#settings . Please also run:

sudo journalctl -u influxdb | grep POST | tail -n20

Which will give you the latest writes to InfluxDB, can you post the output here?

komarEX commented 5 years ago

I'm aware but I want to have all timeseries for each interface (view included). I've already checked if there are any POSTs to InfluxDB - there are none. I can make new database, I can alter settings like retention and it gets into Influx, GET also works but there are no POSTs whatsoever and I don't know why (it all goes into files to ts_export directories).

emanuele-f commented 5 years ago

Please post the output of the following commands:

redis-cli lrange ntopng.influx_file_queue 0 10
redis-cli get ntopng.cache.influxdb_export_time_ntopng_2
redis-cli get ntopng.cache.influxdb_export_time_ntopng_3
redis-cli get ntopng.cache.influxdb_export_time_ntopng_4
komarEX commented 5 years ago 
root@ntopng:~# redis-cli lrange ntopng.influx_file_queue 0 10
 1) "2|1559800894|1161"
 2) "3|1559800889|814"
 3) "4|1559800895|2458"
 4) "2|1559800897|1162"
 5) "3|1559800900|815"
 6) "4|1559800906|2459"
 7) "2|1559800908|1163"
 8) "3|1559800911|816"
 9) "4|1559800917|2460"
10) "2|1559800919|1164"
11) "3|1559800922|817"
root@ntopng:~# redis-cli get ntopng.cache.influxdb_export_time_ntopng_2
"1559779535"
root@ntopng:~# redis-cli get ntopng.cache.influxdb_export_time_ntopng_3
"1559779532"
root@ntopng:~# redis-cli get ntopng.cache.influxdb_export_time_ntopng_4
"1559779312"
emanuele-f commented 5 years ago

Can you manually edit /usr/share/ntopng/scripts/callbacks/system/housekeeping.luaand add tprint("upload ts data") above this line https://github.com/ntop/ntopng/blob/3.8-stable/scripts/callbacks/system/housekeeping.lua#L26 ? Then "upload ts data" should appear into the ntopng log (sudo journalctl -u ntopng -f)

komarEX commented 5 years ago 
cze 07 16:06:26 hostersi-ntopng ntopng[20130]: 07/Jun/2019 16:06:26 [Ntop.cpp:413] Built on Ubuntu 18.04.2 LTS
cze 07 16:06:26 hostersi-ntopng ntopng[20130]: 07/Jun/2019 16:06:26 [NtopPro.cpp:638] [LICENSE] System Id:        8C1263CCB206AB13
cze 07 16:06:26 hostersi-ntopng ntopng[20130]: 07/Jun/2019 16:06:26 [NtopPro.cpp:639] [LICENSE] Edition:        Professional
cze 07 16:06:26 hostersi-ntopng ntopng[20130]: 07/Jun/2019 16:06:26 [NtopPro.cpp:640] [LICENSE] License Type:        Permanent License
cze 07 16:06:26 hostersi-ntopng ntopng[20130]: 07/Jun/2019 16:06:26 [NtopPro.cpp:654] [LICENSE] License Hash:        F88ADEF743DB87DCA088A263F2F71C141575967298FFB63A35
cze 07 16:06:26 hostersi-ntopng ntopng[20130]: 07/Jun/2019 16:06:26 [NtopPro.cpp:664] [LICENSE] Maintenance:        Until Tue Dec 10 09:41:38 2019 [185 days left]
cze 07 16:06:26 hostersi-ntopng ntopng[20130]: 07/Jun/2019 16:06:26 [PeriodicActivities.cpp:72] Started periodic activities loop...
cze 07 16:07:06 hostersi-ntopng ntopng[20130]: 07/Jun/2019 16:07:06 [PeriodicActivities.cpp:113] Each periodic activity script will use 3 threads
cze 07 16:07:06 hostersi-ntopng ntopng[20130]: 07/Jun/2019 16:07:06 [NetworkInterface.cpp:2583] Started packet polling on interface enp2s0f0 [id: 2]...
cze 07 16:07:06 hostersi-ntopng ntopng[20130]: 07/Jun/2019 16:07:06 [NetworkInterface.cpp:2583] Started packet polling on interface enp2s0f1 [id: 3]...
cze 07 16:07:13 hostersi-ntopng ntopng[20130]:  string upload ts data
cze 07 16:07:13 hostersi-ntopng ntopng[20130]: 07/Jun/2019 16:07:13 [LuaEngine.cpp:1713] ERROR: Interface didn't reload custom categories on time [iface: view:enp2s0f0,enp2s0f1]
cze 07 16:16:07 hostersi-ntopng ntopng[20130]:  string upload ts data
cze 07 16:16:07 hostersi-ntopng ntopng[20130]:  string upload ts data

Still I can't see any POSTs to Influx.

emanuele-f commented 5 years ago

This is needed to debug where the problem could be.

Please add tprint(drivers) in /usr/share/ntopng/scripts/lua/modules/callback_utils.lua above this line: https://github.com/ntop/ntopng/blob/89f6544524a2f24bb4739108aef67dbb2d263a02/scripts

and also tprint(name_id) in /usr/share/ntopng/scripts/lua/modules/timeseries/drivers/influxdb.lua above this line: https://github.com/ntop/ntopng/blob/3.8-stable/scripts/lua/modules/timeseries/drivers/influxdb.lua#L463

then post here the log.

komarEX commented 5 years ago

I will do that on Monday :)

komarEX commented 5 years ago

Your first link is broken and doesn't point o exact line.

komarEX commented 5 years ago

Hmm. I've uncommented this line https://github.com/ntop/ntopng/blob/3.8-stable/scripts/lua/modules/timeseries/drivers/influxdb.lua#L489 and threw tprint(name_id) after that. What's funny is that ts_exports directory gets new files but nothing goes into log from ntopng. Is this only place which saves to ts_exports folder?

cze 10 08:36:40 hostersi-ntopng ntopng[16582]: 10/Jun/2019 08:36:40 [NetworkInterface.cpp:2583] Started packet polling on interface enp2s0f0 [id: 2]...
cze 10 08:36:40 hostersi-ntopng ntopng[16582]: 10/Jun/2019 08:36:40 [NetworkInterface.cpp:2583] Started packet polling on interface enp2s0f1 [id: 3]...
cze 10 08:36:47 hostersi-ntopng ntopng[16582]:  string upload ts data
cze 10 08:36:47 hostersi-ntopng ntopng[16582]: 10/Jun/2019 08:36:47 [LuaEngine.cpp:1713] ERROR: Interface didn't reload custom categories on time [iface: view:enp2s0f0,enp2s0f1]
cze 10 08:42:00 hostersi-ntopng ntopng[16582]:  string upload ts data
cze 10 08:42:00 hostersi-ntopng ntopng[16582]:  string upload ts data
cze 10 08:42:00 hostersi-ntopng ntopng[16582]: 10/Jun/2019 08:42:00 [LuaEngine.cpp:7712] [view:enp2s0f0,enp2s0f1] ERROR: Cannot complete top talkers generation in 1 minute. Is there a huge number of hosts in the system?
cze 10 08:43:00 hostersi-ntopng ntopng[16582]: 10/Jun/2019 08:42:00 [minute.lua:32] [rrd_5min_dump_utils.lua:279] ERROR: [view:enp2s0f0,enp2s0f1 ]Cannot complete local hosts RRD dump. Disk slow or too many local hosts?
cze 10 08:43:00 hostersi-ntopng ntopng[16582]:  string upload ts data
cze 10 08:43:00 hostersi-ntopng ntopng[16582]: 10/Jun/2019 08:43:00 [LuaEngine.cpp:7712] [view:enp2s0f0,enp2s0f1] ERROR: Cannot complete top talkers generation in 1 minute. Is there a huge number of hosts in the system?
komarEX commented 5 years ago

Your first link was probably this change:

function callback_utils.uploadTSdata()
   local ts_utils = require("ts_utils_core")
   local drivers = ts_utils.listActiveDrivers()
   tprint(drivers)

   for _, driver in ipairs(drivers) do
      driver:export()
   end
end

Output for that is:

cze 10 08:46:56 hostersi-ntopng ntopng[19985]:  table
cze 10 08:46:56 hostersi-ntopng ntopng[19985]: 1 table
cze 10 08:46:56 hostersi-ntopng ntopng[19985]: 1.password string
cze 10 08:46:56 hostersi-ntopng ntopng[19985]: 1.db string ntopng
cze 10 08:46:56 hostersi-ntopng ntopng[19985]: 1.username string
cze 10 08:46:56 hostersi-ntopng ntopng[19985]: 1.url string http://127.0.0.1:8086
emanuele-f commented 5 years ago

These files are only created by the C core which exports the ts data. This issue is weird, can you please add tprint(name_id) below https://github.com/ntop/ntopng/blob/3.8-stable/scripts/lua/modules/timeseries/drivers/influxdb.lua#L461 and also tprint(ret) below https://github.com/ntop/ntopng/blob/3.8-stable/scripts/lua/modules/timeseries/drivers/influxdb.lua#L482 ?

komarEX commented 5 years ago

I have removed previous tprint lines so only these 2 are present right now. Here's the log:

cze 14 09:11:20 hostersi-ntopng ntopng[7179]: 14/Jun/2019 09:11:20 [PeriodicActivities.cpp:113] Each periodic activity script will use 3 threads
cze 14 09:11:20 hostersi-ntopng ntopng[7179]: 14/Jun/2019 09:11:20 [NetworkInterface.cpp:2583] Started packet polling on interface enp2s0f0 [id: 2]...
cze 14 09:11:20 hostersi-ntopng ntopng[7179]: 14/Jun/2019 09:11:20 [NetworkInterface.cpp:2583] Started packet polling on interface enp2s0f1 [id: 3]...
cze 14 09:11:27 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:11:27 hostersi-ntopng ntopng[7179]:  string 3|1559803980|1095
cze 14 09:11:27 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:11:27 hostersi-ntopng ntopng[7179]: 14/Jun/2019 09:11:27 [LuaEngine.cpp:1713] ERROR: Interface didn't reload custom categories on time [iface: view:enp2s0f0,enp2s0f1]
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559803988|1560
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 4|1559803988|2739
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 3|1559803991|1096
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559803999|1561
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 4|1559803999|2740
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804010|1562
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804011|1563
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 3|1559804002|1097
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804012|1564
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 4|1559804010|2741
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 3|1559804013|1098
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804013|1565
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 4|1559804021|2742
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 3|1559804024|1099
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804024|1566
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 4|1559804032|2743
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 3|1559804035|1100
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804035|1567
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 4|1559804043|2744
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 3|1559804046|1101
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804046|1568
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 4|1559804054|2745
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804057|1569
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 3|1559804057|1102
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804068|1570
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804069|1571
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804070|1572
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 4|1559804065|2746
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]: 14/Jun/2019 09:12:50 [minute.lua:32] [rrd_5min_dump_utils.lua:279] ERROR: [view:enp2s0f0,enp2s0f1 ]Cannot complete local hosts RRD dump. Disk slow or too many local hosts?
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 3|1559804068|1103
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804072|1573
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 4|1559804076|2747
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 3|1559804079|1104
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804083|1574
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 4|1559804087|2748
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 3|1559804090|1105
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804094|1575
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 4|1559804098|2749
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 3|1559804101|1106
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804105|1576
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 4|1559804109|2750
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 3|1559804112|1107
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804116|1577
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 4|1559804120|2751
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804127|1578
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804132|1579
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 3|1559804123|1108
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804134|1580
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 4|1559804131|2752
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 3|1559804134|1109
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 2|1559804136|1581
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string upload ts data
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  string 4|1559804142|2753
cze 14 09:14:00 hostersi-ntopng ntopng[7179]:  nil nil
cze 14 09:14:00 hostersi-ntopng ntopng[7179]: 14/Jun/2019 09:14:00 [LuaEngine.cpp:7712] [view:enp2s0f0,enp2s0f1] ERROR: Cannot complete top talkers generation in 1 minute. Is there a huge number of hosts in the system?
emanuele-f commented 5 years ago

Ok, hopefully this will be the last lines:

tprint("FILE: " .. fname) below https://github.com/ntop/ntopng/blob/3.8-stable/scripts/lua/modules/timeseries/drivers/influxdb.lua#L439

tprint(ret) below https://github.com/ntop/ntopng/blob/3.8-stable/scripts/lua/modules/timeseries/drivers/influxdb.lua#L446

tprint("MSG:" .. msg) below https://github.com/ntop/ntopng/blob/3.8-stable/scripts/lua/modules/timeseries/drivers/influxdb.lua#L448

komarEX commented 5 years ago

I didn't delete 2 previous tprints so we have 5 now in code. Log:

cze 14 10:09:12 hostersi-ntopng ntopng[9740]: 14/Jun/2019 10:09:12 [PeriodicActivities.cpp:113] Each periodic activity script will use 3 threads
cze 14 10:09:12 hostersi-ntopng ntopng[9740]: 14/Jun/2019 10:09:12 [NetworkInterface.cpp:2583] Started packet polling on interface enp2s0f0 [id: 2]...
cze 14 10:09:12 hostersi-ntopng ntopng[9740]: 14/Jun/2019 10:09:12 [NetworkInterface.cpp:2583] Started packet polling on interface enp2s0f1 [id: 3]...
cze 14 10:09:19 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:09:19 hostersi-ntopng ntopng[9740]:  string 2|1559804249|1594
cze 14 10:09:19 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/2/ts_export/1594_1559804249
cze 14 10:09:19 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:09:19 hostersi-ntopng ntopng[9740]: 14/Jun/2019 10:09:19 [LuaEngine.cpp:1713] ERROR: Interface didn't reload custom categories on time [iface: view:enp2s0f0,enp2s0f1]
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 4|1559804241|2762
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/4/ts_export/2762_1559804241
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 3|1559804244|1119
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/3/ts_export/1119_1559804244
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 2|1559804250|1595
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/2/ts_export/1595_1559804250
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 4|1559804252|2763
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/4/ts_export/2763_1559804252
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 3|1559804255|1120
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/3/ts_export/1120_1559804255
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 2|1559804261|1596
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/2/ts_export/1596_1559804261
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 4|1559804263|2764
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/4/ts_export/2764_1559804263
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 3|1559804266|1121
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/3/ts_export/1121_1559804266
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 2|1559804272|1597
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/2/ts_export/1597_1559804272
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 4|1559804274|2765
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/4/ts_export/2765_1559804274
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 3|1559804277|1122
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/3/ts_export/1122_1559804277
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 2|1559804283|1598
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/2/ts_export/1598_1559804283
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 4|1559804285|2766
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/4/ts_export/2766_1559804285
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 3|1559804288|1123
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/3/ts_export/1123_1559804288
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 2|1559804294|1599
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/2/ts_export/1599_1559804294
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 4|1559804296|2767
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/4/ts_export/2767_1559804296
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 2|1559804305|1600
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/2/ts_export/1600_1559804305
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 3|1559804299|1124
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/3/ts_export/1124_1559804299
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 2|1559804309|1601
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/2/ts_export/1601_1559804309
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 2|1559804310|1602
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/2/ts_export/1602_1559804310
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 4|1559804307|2768
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/4/ts_export/2768_1559804307
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 3|1559804310|1125
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/3/ts_export/1125_1559804310
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 2|1559804312|1603
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/2/ts_export/1603_1559804312
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 4|1559804318|2769
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/4/ts_export/2769_1559804318
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 3|1559804321|1126
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/3/ts_export/1126_1559804321
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 2|1559804323|1604
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/2/ts_export/1604_1559804323
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 4|1559804329|2770
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/4/ts_export/2770_1559804329
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 3|1559804332|1127
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/3/ts_export/1127_1559804332
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 2|1559804334|1605
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/2/ts_export/1605_1559804334
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 4|1559804340|2771
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/4/ts_export/2771_1559804340
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 3|1559804343|1128
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/3/ts_export/1128_1559804343
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 2|1559804345|1606
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/2/ts_export/1606_1559804345
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 4|1559804351|2772
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/4/ts_export/2772_1559804351
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 3|1559804354|1129
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/3/ts_export/1129_1559804354
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string 2|1559804356|1607
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string FILE: /var/log/ntopng/2/ts_export/1607_1559804356
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  nil nil
cze 14 10:11:04 hostersi-ntopng ntopng[9740]:  string upload ts data
emanuele-f commented 5 years ago

Please remove the old ones. Can you also run ls -l /var/log/ntopng/4/ts_export/2762_1559804241?

komarEX commented 5 years ago

I've stopped ntopng after that log and for some reason I don't have these files. Only couple of new:

# ls -la /var/log/ntopng/4/ts_export/
total 3980
drwx------ 2 ntopng ntopng    4096 cze 14 10:12 .
drwx------ 6 ntopng ntopng      67 cze 14 10:09 ..
-rw------- 1 ntopng ntopng    1866 cze 14 10:09 0_1560499762
-rw------- 1 ntopng ntopng    1859 cze 14 10:11 10_1560499872
-rw------- 1 ntopng ntopng    1859 cze 14 10:11 11_1560499883
-rw------- 1 ntopng ntopng    1837 cze 14 10:09 1_1560499773
-rw------- 1 ntopng ntopng    1859 cze 14 10:11 12_1560499894
-rw------- 1 ntopng ntopng    1859 cze 14 10:11 13_1560499905
-rw------- 1 ntopng ntopng    1875 cze 14 10:11 14_1560499916
-rw------- 1 ntopng ntopng 1200694 cze 14 10:12 15_1560499927
-rw------- 1 ntopng ntopng    1881 cze 14 10:12 16_1560499938
-rw------- 1 ntopng ntopng    1881 cze 14 10:12 17_1560499949
-rw------- 1 ntopng ntopng    1881 cze 14 10:12 18_1560499960
-rw------- 1 ntopng ntopng    1881 cze 14 10:12 19_1560499971
-rw------- 1 ntopng ntopng  871319 cze 14 10:13 20_1560499982
-rw------- 1 ntopng ntopng    1838 cze 14 10:09 2_1560499784
-rw------- 1 ntopng ntopng    1852 cze 14 10:09 3_1560499795
-rw------- 1 ntopng ntopng    1859 cze 14 10:10 4_1560499806
-rw------- 1 ntopng ntopng    1859 cze 14 10:10 5_1560499817
-rw------- 1 ntopng ntopng    1859 cze 14 10:10 6_1560499828
-rw------- 1 ntopng ntopng    1859 cze 14 10:10 7_1560499839
-rw------- 1 ntopng ntopng  910493 cze 14 10:10 8_1560499850
-rw------- 1 ntopng ntopng 1008767 cze 14 10:11 9_1560499861

I will remove older tprints and leave only 3 newest from https://github.com/ntop/ntopng/issues/2643#issuecomment-502008771

Log:

cze 14 11:12:53 hostersi-ntopng ntopng[12298]: 14/Jun/2019 11:12:53 [PeriodicActivities.cpp:113] Each periodic activity script will use 3 threads
cze 14 11:12:53 hostersi-ntopng ntopng[12298]: 14/Jun/2019 11:12:53 [NetworkInterface.cpp:2583] Started packet polling on interface enp2s0f0 [id: 2]...
cze 14 11:12:53 hostersi-ntopng ntopng[12298]: 14/Jun/2019 11:12:53 [NetworkInterface.cpp:2583] Started packet polling on interface enp2s0f1 [id: 3]...
cze 14 11:12:59 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:12:59 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1625_1559804492
cze 14 11:12:59 hostersi-ntopng ntopng[12298]: 14/Jun/2019 11:12:59 [LuaEngine.cpp:1713] ERROR: Interface didn't reload custom categories on time [iface: view:enp2s0f0,enp2s0f1]
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/3/ts_export/1141_1559804486
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/4/ts_export/2785_1559804494
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1626_1559804494
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/3/ts_export/1142_1559804497
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1627_1559804505
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/4/ts_export/2786_1559804505
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/3/ts_export/1143_1559804508
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1628_1559804516
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/4/ts_export/2787_1559804516
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/3/ts_export/1144_1559804519
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1629_1559804527
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/4/ts_export/2788_1559804527
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/3/ts_export/1145_1559804530
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1630_1559804538
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/4/ts_export/2789_1559804538
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1631_1559804549
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1632_1559804550
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/3/ts_export/1146_1559804541
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/4/ts_export/2790_1559804549
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1633_1559804551
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/3/ts_export/1147_1559804552
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/4/ts_export/2791_1559804560
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1634_1559804562
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/3/ts_export/1148_1559804563
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/4/ts_export/2792_1559804571
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1635_1559804573
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/3/ts_export/1149_1559804574
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/4/ts_export/2793_1559804582
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1636_1559804584
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/3/ts_export/1150_1559804585
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/4/ts_export/2794_1559804593
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1637_1559804595
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/3/ts_export/1151_1559804596
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1638_1559804606
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1639_1559804610
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1640_1559804611
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/4/ts_export/2795_1559804604
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/3/ts_export/1152_1559804607
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1641_1559804613
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/4/ts_export/2796_1559804615
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/3/ts_export/1153_1559804618
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1642_1559804624
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/4/ts_export/2797_1559804626
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/3/ts_export/1154_1559804629
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1643_1559804635
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/4/ts_export/2798_1559804637
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/3/ts_export/1155_1559804640
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/2/ts_export/1644_1559804646
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/4/ts_export/2799_1559804648
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string FILE: /var/log/ntopng/3/ts_export/1156_1559804651
cze 14 11:15:29 hostersi-ntopng ntopng[12298]:  string upload ts data

Directories (after that log generation):

# ls -la /var/log/ntopng/*/ts_export/
/var/log/ntopng/2/ts_export/:
total 3036
drwx------ 2 ntopng ntopng   4096 cze 14 11:17 .
drwx------ 7 ntopng ntopng     86 cze 14 11:12 ..
-rw------- 1 ntopng ntopng 476635 cze 14 11:13 0_1560503583
-rw------- 1 ntopng ntopng   1815 cze 14 11:14 10_1560503693
-rw------- 1 ntopng ntopng 626372 cze 14 11:15 11_1560503704
-rw------- 1 ntopng ntopng   1793 cze 14 11:13 1_1560503594
-rw------- 1 ntopng ntopng   1815 cze 14 11:15 12_1560503715
-rw------- 1 ntopng ntopng   1815 cze 14 11:15 13_1560503726
-rw------- 1 ntopng ntopng   1815 cze 14 11:15 14_1560503737
-rw------- 1 ntopng ntopng   1815 cze 14 11:15 15_1560503748
-rw------- 1 ntopng ntopng   1815 cze 14 11:16 16_1560503759
-rw------- 1 ntopng ntopng 659061 cze 14 11:16 17_1560503770
-rw------- 1 ntopng ntopng   1815 cze 14 11:16 18_1560503781
-rw------- 1 ntopng ntopng   1815 cze 14 11:16 19_1560503792
-rw------- 1 ntopng ntopng   1815 cze 14 11:16 20_1560503803
-rw------- 1 ntopng ntopng   1815 cze 14 11:16 21_1560503814
-rw------- 1 ntopng ntopng   1800 cze 14 11:13 2_1560503605
-rw------- 1 ntopng ntopng 687604 cze 14 11:17 22_1560503825
-rw------- 1 ntopng ntopng    768 cze 14 11:17 23_1560503836
-rw------- 1 ntopng ntopng   1804 cze 14 11:13 3_1560503616
-rw------- 1 ntopng ntopng   1810 cze 14 11:13 4_1560503627
-rw------- 1 ntopng ntopng   1815 cze 14 11:13 5_1560503638
-rw------- 1 ntopng ntopng 571007 cze 14 11:14 6_1560503649
-rw------- 1 ntopng ntopng   1815 cze 14 11:14 7_1560503660
-rw------- 1 ntopng ntopng   1815 cze 14 11:14 8_1560503671
-rw------- 1 ntopng ntopng   1815 cze 14 11:14 9_1560503682

/var/log/ntopng/3/ts_export/:
total 6220
drwx------ 2 ntopng ntopng    4096 cze 14 11:17 .
drwx------ 6 ntopng ntopng      67 cze 14 11:12 ..
-rw------- 1 ntopng ntopng  891885 cze 14 11:13 0_1560503583
-rw------- 1 ntopng ntopng    1848 cze 14 11:14 10_1560503693
-rw------- 1 ntopng ntopng 1299048 cze 14 11:15 11_1560503704
-rw------- 1 ntopng ntopng    1826 cze 14 11:13 1_1560503594
-rw------- 1 ntopng ntopng    1848 cze 14 11:15 12_1560503715
-rw------- 1 ntopng ntopng    1868 cze 14 11:15 13_1560503726
-rw------- 1 ntopng ntopng    1870 cze 14 11:15 14_1560503737
-rw------- 1 ntopng ntopng    1870 cze 14 11:15 15_1560503748
-rw------- 1 ntopng ntopng    1870 cze 14 11:16 16_1560503759
-rw------- 1 ntopng ntopng 1409101 cze 14 11:16 17_1560503770
-rw------- 1 ntopng ntopng    1870 cze 14 11:16 18_1560503781
-rw------- 1 ntopng ntopng    1870 cze 14 11:16 19_1560503792
-rw------- 1 ntopng ntopng    1870 cze 14 11:16 20_1560503803
-rw------- 1 ntopng ntopng    1870 cze 14 11:16 21_1560503814
-rw------- 1 ntopng ntopng    1832 cze 14 11:13 2_1560503605
-rw------- 1 ntopng ntopng 1514130 cze 14 11:17 22_1560503825
-rw------- 1 ntopng ntopng     790 cze 14 11:17 23_1560503836
-rw------- 1 ntopng ntopng    1848 cze 14 11:13 3_1560503616
-rw------- 1 ntopng ntopng    1848 cze 14 11:13 4_1560503627
-rw------- 1 ntopng ntopng    1848 cze 14 11:13 5_1560503638
-rw------- 1 ntopng ntopng 1161432 cze 14 11:14 6_1560503649
-rw------- 1 ntopng ntopng    1848 cze 14 11:14 7_1560503660
-rw------- 1 ntopng ntopng    1848 cze 14 11:14 8_1560503671
-rw------- 1 ntopng ntopng    1848 cze 14 11:14 9_1560503682

/var/log/ntopng/4/ts_export/:
total 3168
drwx------ 2 ntopng ntopng    4096 cze 14 11:17 .
drwx------ 6 ntopng ntopng      67 cze 14 11:12 ..
-rw------- 1 ntopng ntopng    1884 cze 14 11:13 0_1560503583
-rw------- 1 ntopng ntopng    1859 cze 14 11:14 10_1560503693
-rw------- 1 ntopng ntopng 1262300 cze 14 11:15 11_1560503704
-rw------- 1 ntopng ntopng  803466 cze 14 11:13 1_1560503594
-rw------- 1 ntopng ntopng    1873 cze 14 11:15 12_1560503715
-rw------- 1 ntopng ntopng    1881 cze 14 11:15 13_1560503726
-rw------- 1 ntopng ntopng    1881 cze 14 11:15 14_1560503737
-rw------- 1 ntopng ntopng    1881 cze 14 11:15 15_1560503748
-rw------- 1 ntopng ntopng    1881 cze 14 11:16 16_1560503759
-rw------- 1 ntopng ntopng    1881 cze 14 11:16 17_1560503770
-rw------- 1 ntopng ntopng    1881 cze 14 11:16 18_1560503781
-rw------- 1 ntopng ntopng    1881 cze 14 11:16 19_1560503792
-rw------- 1 ntopng ntopng    1881 cze 14 11:16 20_1560503803
-rw------- 1 ntopng ntopng    1881 cze 14 11:16 21_1560503814
-rw------- 1 ntopng ntopng    1847 cze 14 11:13 2_1560503605
-rw------- 1 ntopng ntopng    8080 cze 14 11:17 22_1560503825
-rw------- 1 ntopng ntopng     795 cze 14 11:17 23_1560503836
-rw------- 1 ntopng ntopng    1859 cze 14 11:13 3_1560503616
-rw------- 1 ntopng ntopng    1859 cze 14 11:13 4_1560503627
-rw------- 1 ntopng ntopng    1859 cze 14 11:13 5_1560503638
-rw------- 1 ntopng ntopng 1075123 cze 14 11:14 6_1560503649
-rw------- 1 ntopng ntopng    1859 cze 14 11:14 7_1560503660
-rw------- 1 ntopng ntopng    1859 cze 14 11:14 8_1560503671
-rw------- 1 ntopng ntopng    1859 cze 14 11:14 9_1560503682

For some reason these files don't exist there.

Epoch time in saved files is more or less real but epoch time which shows in log is from about 8 days ago.

Last data I see in influx (based on graphs) is from about 8 days ago. There's some time inconsistency in Redis/InfluxDB data? I'm not sure if that is expected.

komarEX commented 5 years ago

Maybe this strace will help you a bit:

# strace -e trace=file -f -p 12298 2>&1 | grep ts_export
[pid 12346] openat(AT_FDCWD, "/var/log/ntopng/3/ts_export/81_1560504474", O_RDWR|O_CREAT|O_EXCL|O_NOCTTY, 0600) = 21
[pid 12346] openat(AT_FDCWD, "/var/log/ntopng/4/ts_export/81_1560504474", O_RDWR|O_CREAT|O_EXCL|O_NOCTTY, 0600) = 16
[pid 12346] openat(AT_FDCWD, "/var/log/ntopng/2/ts_export/81_1560504474", O_RDWR|O_CREAT|O_EXCL|O_NOCTTY, 0600) = 22
[pid 12361] stat("/var/log/ntopng/2/ts_export/1748_1559805449", 0x7fc69dffa1d0) = -1 ENOENT (No such file or directory)
[pid 12361] stat("/var/log/ntopng/4/ts_export/2871_1559805440", 0x7fc69dffa1d0) = -1 ENOENT (No such file or directory)
[pid 12361] stat("/var/log/ntopng/2/ts_export/1749_1559805450", 0x7fc69dffa1d0) = -1 ENOENT (No such file or directory)
[pid 12361] stat("/var/log/ntopng/3/ts_export/1228_1559805443", 0x7fc69dffa1d0) = -1 ENOENT (No such file or directory)
[pid 12346] openat(AT_FDCWD, "/var/log/ntopng/4/ts_export/82_1560504485", O_RDWR|O_CREAT|O_EXCL|O_NOCTTY, 0600) = 16
[pid 12346] openat(AT_FDCWD, "/var/log/ntopng/2/ts_export/82_1560504485", O_RDWR|O_CREAT|O_EXCL|O_NOCTTY, 0600) = 22
[pid 12346] openat(AT_FDCWD, "/var/log/ntopng/3/ts_export/82_1560504485", O_RDWR|O_CREAT|O_EXCL|O_NOCTTY, 0600) = 21
[pid 12361] stat("/var/log/ntopng/4/ts_export/2872_1559805451", 0x7fc69dffa1d0) = -1 ENOENT (No such file or directory)
[pid 12361] stat("/var/log/ntopng/2/ts_export/1750_1559805451", 0x7fc69dffa1d0) = -1 ENOENT (No such file or directory)
[pid 12361] stat("/var/log/ntopng/3/ts_export/1229_1559805454", 0x7fc69dffa1d0) = -1 ENOENT (No such file or directory)
[pid 12361] stat("/var/log/ntopng/2/ts_export/1751_1559805462", 0x7fc69dffa1d0) = -1 ENOENT (No such file or directory)
[pid 12346] openat(AT_FDCWD, "/var/log/ntopng/3/ts_export/83_1560504496", O_RDWR|O_CREAT|O_EXCL|O_NOCTTY, 0600) = 21
[pid 12346] openat(AT_FDCWD, "/var/log/ntopng/2/ts_export/83_1560504496", O_RDWR|O_CREAT|O_EXCL|O_NOCTTY, 0600) = 22
[pid 12346] openat(AT_FDCWD, "/var/log/ntopng/4/ts_export/83_1560504496", O_RDWR|O_CREAT|O_EXCL|O_NOCTTY, 0600) = 16
[pid 12361] stat("/var/log/ntopng/4/ts_export/2873_1559805462", 0x7fc69dffa1d0) = -1 ENOENT (No such file or directory)
[pid 12361] stat("/var/log/ntopng/3/ts_export/1230_1559805465", 0x7fc69dffa1d0) = -1 ENOENT (No such file or directory)
[pid 12361] stat("/var/log/ntopng/4/ts_export/2874_1559805473", 0x7fc69dffa1d0) = -1 ENOENT (No such file or directory)
[pid 12346] openat(AT_FDCWD, "/var/log/ntopng/4/ts_export/84_1560504507", O_RDWR|O_CREAT|O_EXCL|O_NOCTTY, 0600) = 16
[pid 12346] openat(AT_FDCWD, "/var/log/ntopng/2/ts_export/84_1560504507", O_RDWR|O_CREAT|O_EXCL|O_NOCTTY, 0600) = 22
[pid 12346] openat(AT_FDCWD, "/var/log/ntopng/3/ts_export/84_1560504507", O_RDWR|O_CREAT|O_EXCL|O_NOCTTY, 0600) = 21
[pid 12361] stat("/var/log/ntopng/2/ts_export/1752_1559805473", 0x7fc69dffa1d0) = -1 ENOENT (No such file or directory)
[pid 12361] stat("/var/log/ntopng/3/ts_export/1231_1559805476", 0x7fc69dffa1d0) = -1 ENOENT (No such file or directory)
[pid 12361] stat("/var/log/ntopng/2/ts_export/1753_1559805484", 0x7fc69dffa1d0) = -1 ENOENT (No such file or directory)
[pid 12361] stat("/var/log/ntopng/4/ts_export/2875_1559805484", 0x7fc69dffa1d0) = -1 ENOENT (No such file or directory)
[pid 12346] openat(AT_FDCWD, "/var/log/ntopng/3/ts_export/85_1560504518", O_RDWR|O_CREAT|O_EXCL|O_NOCTTY, 0600) = 21
[pid 12346] openat(AT_FDCWD, "/var/log/ntopng/2/ts_export/85_1560504518", O_RDWR|O_CREAT|O_EXCL|O_NOCTTY, 0600) = 22
[pid 12346] openat(AT_FDCWD, "/var/log/ntopng/4/ts_export/85_1560504518", O_RDWR|O_CREAT|O_EXCL|O_NOCTTY, 0600) = 16
[pid 12361] stat("/var/log/ntopng/3/ts_export/1232_1559805487", 0x7fc69dffa1d0) = -1 ENOENT (No such file or directory)
[pid 12361] stat("/var/log/ntopng/2/ts_export/1754_1559805495", 0x7fc69dffa1d0) = -1 ENOENT (No such file or directory)
[pid 12361] stat("/var/log/ntopng/4/ts_export/2876_1559805495", 0x7fc69dffa1d0) = -1 ENOENT (No such file or directory)

Can you confirm something with me? Does ntopng expected these old (8 days old) files to exist? We had issue with ntopng saving a lot of files and if I remember correctly someone might have deleted these old files to "save space". Still new files are being generated and nothing goes into InfluxDB anyway.

emanuele-f commented 5 years ago

The problem could be exactly that, ntopng expects that files to exist and for some reason the export queue got a lot of unexported files. Please try to manually clean the export queue with redis-cli del ntopng.influx_file_queue.

komarEX commented 5 years ago

It was exactly that. After I purged influx_file_queue everything went back to normal. Essentially I don't think that's proper way to handle it with this kind of behavior. I propose to change it to "if ntopng expects file to exist it will print so in log but it WON'T stop parsing any other ts_export files". Generally speaking because of this issue we have lost about 2 weeks of network data because ntopng was turned off. It happened - ok, but I think it should be changed so something like that won't happen again.

emanuele-f commented 5 years ago

I fully agree with this, please feel free to comment https://github.com/ntop/ntopng/issues/2659 with additional suggestions.