search

ntop / ntopng

Web-based Traffic and Security Network Traffic Monitoring
http://www.ntop.org
GNU General Public License v3.0
6.28k stars 656 forks source link

Traffic Dashboard not showing when using InfluxDB #2527

Closed dive76 closed 5 years ago

dive76 commented 5 years ago

Traffic Dashboard is not showing when using InfluxDB for time series.

I have done some more steps to find the issue:

  1. Switch to RRD: In RRD all works well and traffic dashboard works
  2. Delete ntopng database in InfluxDB
  3. Switch back to InfluxDB. I get a message that the database is successfully created and I can also see this in the admin of InfluxDB: Problem dashboard does not show any traffic.

I get the following error in the log:

06/Apr/2019 10:41:00 [LuaEngine.cpp:8737] WARNING: Script failure [/usr/share/ntopng/scripts/callbacks/system/housekeeping.lua][...opng/scripts/lua/modules/timeseries/drivers/influxdb.lua:616: attempt to index a number value (local 'ret')]

Looks to me like some issue between InfluxDB and Ntopng.

v.3.9.190410 [Enterprise/Professional build] GIT rev: dev:faf8819a9d476908fd84460824cc18923437600f:20190410 Pro rev: r2310 Built on: Ubuntu 18.04.2 LTS System Id: 7908A22C920BAB21 Platform: x86_64 Edition: Enterprise License Type: Permanent 2019-04-10

emanuele-f commented 5 years ago

The WARNING log date is 6 April, do you still see that warning with current version? Do you have InfluxDB Export Failure alerts in ntopng? If you go into the network interface charts page (https://www.ntop.org/guides/ntopng/web_gui/historical.html), can you see charts properly populated?

dive76 commented 5 years ago

The warning does not show anymore. Charts on network interface are showing correctly for 5m and 30m and 1h. Charts from 1 day onward do not show.

emanuele-f commented 5 years ago

Please note that previous data from RRD will not be migrated to InfluxDB, so it's normal that you can only see data starting from the time InfluxDB export was activated.

dive76 commented 5 years ago

I have now waited for more than 24 hours. Result is still the same. Traffic Dashboard and Interface no data. 2019-04-12

emanuele-f commented 5 years ago

Can you post the full log of ntopng since startup? Can you also follow the instructions for https://github.com/ntop/ntopng/blob/dev/doc/README.beta_features#L19 ? A new dropdown will appear in that page where you can select "day" "hour" or "raw". Please check out if selecting "raw" actually shows the points

dive76 commented 5 years ago

image

Enabled the feature and raw shows the data points.

Where can I find the full log file?

emanuele-f commented 5 years ago

So this is a continuous queries problem. Do you run ntopng continuosly or stop it every day? You can get the log via the command sudo journalctl -u ntopng then scroll to the bottom of the log until last startup time

dive76 commented 5 years ago

I run it continuously but update regularly:

Apr 15 21:33:41 gateway systemd[1]: Started ntopng high-speed web-based traffic monitoring and analysis tool. Apr 15 21:33:41 gateway ntopng[29542]: 15/Apr/2019 21:33:41 [Ntop.cpp:1969] Setting local networks to 192.168.0.0/24 Apr 15 21:33:41 gateway ntopng[29542]: 15/Apr/2019 21:33:41 [Redis.cpp:152] Successfully connected to redis 127.0.0.1:6379@0 Apr 15 21:33:41 gateway ntopng[29542]: 15/Apr/2019 21:33:41 [Redis.cpp:152] Successfully connected to redis 127.0.0.1:6379@0 Apr 15 21:33:41 gateway ntopng[29542]: 15/Apr/2019 21:33:41 [NtopPro.cpp:314] [LICENSE] Reading license from Redis Apr 15 21:33:41 gateway ntopng[29542]: 15/Apr/2019 21:33:41 [NtopPro.cpp:208] [LICENSE] B61BD985353E360094BD292D94589E5F1583366827D8B20FEE: found valid E Apr 15 21:33:44 gateway ntopng[29542]: 15/Apr/2019 21:33:44 [PF_RINGInterface.cpp:53] Reading packets from PF_RING v.7.5.0 interface enp1s0... Apr 15 21:33:44 gateway ntopng[29542]: 15/Apr/2019 21:33:44 [Ntop.cpp:2061] Registered interface enp1s0 [id: 0] Apr 15 21:33:44 gateway ntopng[29542]: 15/Apr/2019 21:33:44 [PF_RINGInterface.cpp:53] Reading packets from PF_RING v.7.5.0 interface bond0... Apr 15 21:33:44 gateway ntopng[29542]: 15/Apr/2019 21:33:44 [Ntop.cpp:2061] Registered interface bond0 [id: 7] Apr 15 21:33:44 gateway ntopng[29542]: 15/Apr/2019 21:33:44 [main.cpp:304] PID stored in file /var/run/ntopng.pid Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [Utils.cpp:655] User changed to ntopng Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [HTTPserver.cpp:1327] Web server dirs [/usr/share/ntopng/httpdocs][/usr/share/ntopng/scripts] Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [HTTPserver.cpp:1330] HTTP server listening on 3000 Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [main.cpp:379] Working directory: /var/lib/ntopng Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [main.cpp:381] Scripts/HTML pages directory: /usr/share/ntopng Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [Ntop.cpp:429] Welcome to ntopng x86_64 v.3.9.190415 - (C) 1998-19 ntop.org Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [Ntop.cpp:439] Built on Ubuntu 18.04.2 LTS Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [NtopPro.cpp:650] [LICENSE] System Id: 7908A22C920BAB21 Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [NtopPro.cpp:651] [LICENSE] Edition: Enterprise Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [NtopPro.cpp:652] [LICENSE] License Type: Permanent License Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [NtopPro.cpp:666] [LICENSE] License Hash: B61BD985353E360094BD292D94589E5F1583366827D8 Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [NtopPro.cpp:676] [LICENSE] Maintenance: Until Thu Mar 5 08:07:07 2020 [324 days left Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [Ntop.cpp:764] Adding 192.168.0.46/32 as IPv4 interface address for enp1s0 Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [Ntop.cpp:772] Adding 192.168.0.0/24 as IPv4 local network for enp1s0 Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [Ntop.cpp:792] Adding 2406:3003:2006:2d28:2e0:67ff:fe05:9c2a/128 as IPv6 interface address fo Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [Ntop.cpp:801] Adding 2406:3003:2006:2d28:2e0:67ff:fe05:9c2a/64 as IPv6 local network for enp Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [Ntop.cpp:792] Adding fe80::2e0:67ff:fe05:9c2a/128 as IPv6 interface address for enp1s0 Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [Ntop.cpp:801] Adding fe80::2e0:67ff:fe05:9c2a/64 as IPv6 local network for enp1s0 Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [Ntop.cpp:792] Adding 2406:3003:2006:2d28:acd9:4e10:833f:25bb/128 as IPv6 interface address f Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [Ntop.cpp:801] Adding 2406:3003:2006:2d28:acd9:4e10:833f:25bb/64 as IPv6 local network for bo Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [Ntop.cpp:792] Adding 2406:3003:2006:2d28:2e0:67ff:fe05:9c2c/128 as IPv6 interface address fo Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [Ntop.cpp:801] Adding 2406:3003:2006:2d28:2e0:67ff:fe05:9c2c/64 as IPv6 local network for bon Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [Ntop.cpp:792] Adding fe80::2e0:67ff:fe05:9c2c/128 as IPv6 interface address for bond0 Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [Ntop.cpp:801] Adding fe80::2e0:67ff:fe05:9c2c/64 as IPv6 local network for bond0 Apr 15 21:33:45 gateway ntopng[29542]: 15/Apr/2019 21:33:45 [PeriodicActivities.cpp:72] Started periodic activities loop... Apr 15 21:33:47 gateway ntopng[29542]: 15/Apr/2019 21:33:47 [PeriodicActivities.cpp:113] Each periodic activity script will use 2 threads Apr 15 21:33:47 gateway ntopng[29542]: 15/Apr/2019 21:33:47 [NetworkInterface.cpp:2700] Started packet polling on interface enp1s0 [id: 0]... Apr 15 21:33:47 gateway ntopng[29542]: 15/Apr/2019 21:33:47 [NetworkInterface.cpp:2700] Started packet polling on interface bond0 [id: 7]... lines 85721-85766/85766 (END)

emanuele-f commented 5 years ago

Can you inspect the output of sudo journalctl -u influxdb | grep continuous_querier_execute | less on the InfluxDB server to see if there are any reported errors?

dive76 commented 5 years ago

Found no errors:

Apr 16 12:00:00 gateway influxd[1582]: ts=2019-04-16T04:00:00.486984Z lvl=info msg="Finished continuous query" log_id=0EYACNP0000 service=continuous_querier trace_id=0Eq4fNuW000 op_name=continuous_querier_execute name=host:1d_delta_traffic_volume1h db_instance=ntopng written=0 start=2019-04-16T02:00:00.000000Z end=2019-04-16T04:00:00.000000Z duration=0ms Apr 16 12:00:00 gateway influxd[1582]: ts=2019-04-16T04:00:00.487125Z lvl=info msg="Continuous query execution (end)" log_id=0EYACNP0000 service=continuous_querier trace_id=0Eq4fNuW000 op_name=continuous_querier_execute op_event=end op_elapsed=0.978ms Apr 16 12:00:00 gateway influxd[1582]: ts=2019-04-16T04:00:00.487454Z lvl=info msg="Continuous query execution (start)" log_id=0EYACNP0000 service=continuous_querier trace_id=0Eq4fNul000 op_name=continuous_querier_execute op_event=start Apr 16 12:00:00 gateway influxd[1582]: ts=2019-04-16T04:00:00.487566Z lvl=info msg="Executing continuous query" log_id=0EYACNP0000 service=continuous_querier trace_id=0Eq4fNul000 op_name=continuous_querier_execute name=country:traffic1h db_instance=ntopng start=2019-04-16T02:00:00.000000Z end=2019-04-16T04:00:00.000000Z Apr 16 12:00:00 gateway influxd[1582]: ts=2019-04-16T04:00:00.487979Z lvl=info msg="Continuous query execution (end)" log_id=0EYACNP0000 service=continuous_querier trace_id=0Eq4fNul000 op_name=continuous_querier_execute op_event=end op_elapsed=0.529ms Apr 16 12:00:00 gateway influxd[1582]: ts=2019-04-16T04:00:00.488389Z lvl=info msg="Continuous query execution (start)" log_id=0EYACNP0000 service=continuous_querier trace_id=0Eq4fNv0000 op_name=continuous_querier_execute op_event=start Apr 16 12:00:00 gateway influxd[1582]: ts=2019-04-16T04:00:00.488501Z lvl=info msg="Executing continuous query" log_id=0EYACNP0000 service=continuous_querier trace_id=0Eq4fNv0000 op_name=continuous_querier_execute name=iface:1d_delta_flows__1h db_instance=ntopng start=2019-04-16T02:00:00.000000Z end=2019-04-16T04:00:00.000000Z

emanuele-f commented 5 years ago

If you filter by iface:traffic__1h (which is the timeseries of the empty chart) do you see errors? sudo journalctl -u influxdb | grep continuous_querier_execute | grep iface:traffic__1h | less

dive76 commented 5 years ago

No errors:

log_id=0EYACNP0000 service=continuous_querier trace_id=0EpoVl5l000 op_name=continuous_querier_execute name=iface:traffic1h db_instance=ntopng start=2019-04-15T21:00:00.000000Z end=2019-04-15T23:00:00.000000Z Apr 16 08:00:01 gateway influxd[1582]: ts=2019-04-16T00:00:01.336264Z lvl=info msg="Executing continuous query" log_id=0EYACNP0000 service=continuous_querier trace_id=0EprwXE0000 op_name=continuous_querier_execute name=iface:traffic__1h db_instance=ntopng start=2019-04-15T22:00:00.000000Z end=2019-04-16T00:00:00.000000Z Apr 16 09:00:00 gateway influxd[1582]: ts=2019-04-16T01:00:00.381138Z lvl=info msg="Executing continuous query" log_id=0EYACNP0000 service=continuous_querier trace_id=0EpvNB~G000 op_name=continuous_querier_execute name=iface:traffic1h db_instance=ntopng start=2019-04-15T23:00:00.000000Z end=2019-04-16T01:00:00.000000Z Apr 16 10:00:00 gateway influxd[1582]: ts=2019-04-16T02:00:00.586483Z lvl=info msg="Executing continuous query" log_id=0EYACNP0000 service=continuous_querier trace_id=0EpynwIW000 op_name=continuous_querier_execute name=iface:traffic1h db_instance=ntopng start=2019-04-16T00:00:00.000000Z end=2019-04-16T02:00:00.000000Z Apr 16 11:00:00 gateway influxd[1582]: ts=2019-04-16T03:00:00.350908Z lvl=info msg="Executing continuous query" log_id=0EYACNP0000 service=continuous_querier trace_id=0Eq1EdsW000 op_name=continuous_querier_execute name=iface:traffic__1h db_instance=ntopng start=2019-04-16T01:00:00.000000Z end=2019-04-16T03:00:00.000000Z Apr 16 12:00:00 gateway influxd[1582]: ts=2019-04-16T04:00:00.398689Z lvl=info msg="Executing continuous query" log_id=0EYACNP0000 service=continuous_querier trace_id=0Eq4fNZW002 op_name=continuous_querier_execute name=iface:traffic1h db_instance=ntopng start=2019-04-16T02:00:00.000000Z end=2019-04-16T04:00:00.000000Z Apr 16 13:00:00 gateway influxd[1582]: ts=2019-04-16T05:00:00.661563Z lvl=info msg="Executing continuous query" log_id=0EYACNP0000 service=continuous_querier trace_id=0Eq8675G000 op_name=continuous_querier_execute name=iface:traffic1h db_instance=ntopng start=2019-04-16T03:00:00.000000Z end=2019-04-16T05:00:00.000000Z Apr 16 14:00:00 gateway influxd[1582]: ts=2019-04-16T06:00:00.943132Z lvl=info msg="Executing continuous query" log_id=0EYACNP0000 service=continuous_querier trace_id=0EqBXrgl000 op_name=continuous_querier_execute name=iface:traffic__1h db_instance=ntopng start=2019-04-16T04:00:00.000000Z end=2019-04-16T06:00:00.000000Z Apr 16 15:00:01 gateway influxd[1582]: ts=2019-04-16T07:00:01.074346Z lvl=info msg="Executing continuous query" log_id=0EYACNP0000 service=continuous_querier trace_id=0EqEyahW000 op_name=continuous_querier_execute name=iface:traffic1h db_instance=ntopng start=2019-04-16T05:00:00.000000Z end=2019-04-16T07:00:00.000000Z Apr 16 16:00:01 gateway influxd[1582]: ts=2019-04-16T08:00:01.122899Z lvl=info msg="Executing continuous query" log_id=0EYACNP0000 service=continuous_querier trace_id=0EqIPKOW000 op_name=continuous_querier_execute name=iface:traffic__1h db_instance=ntopng start=2019-04-16T06:00:00.000000Z end=2019-04-16T08:00:00.000000Z

emanuele-f commented 5 years ago

Please execute the following command in the influxdb server:

influx -database ntopng -execute "select bytes from \"1h\".\"iface:traffic\" ORDER BY time desc limit 10"

and post the output. Change the "ntopng" database according to your configuration

dive76 commented 5 years ago

It did only get the following (no extra output):

lars@gateway:~$ influx -database ntopng -execute "select bytes from \"1h\".\"iface:traffic\" ORDER BY time desc limit 10" lars@gateway:~$

ntopng is the correct database

emanuele-f commented 5 years ago

Which influxdb version are you using? Please also post the output of:

influx -database ntopng -execute "show continuous queries" | grep "iface:traffic__1h"
influx -database ntopng -execute "show retention policies"

Can you see any other chart with "1h" resolution selected from the dropdown on the right?

dive76 commented 5 years ago

influxdb version is 1.6.1

lars@gateway:~$ influx -database ntopng -execute "show continuous queries" | grep "iface:traffic1h" iface:traffic1h CREATE CONTINUOUS QUERY "iface:traffic__1h" ON ntopng RESAMPLE FOR 2h BEGIN SELECT (sum(bytes) / 3600) AS bytes INTO ntopng."1h"."iface:traffic" FROM (SELECT non_negative_difference(bytes) AS bytes FROM ntopng.autogen."iface:traffic") GROUP BY time(1h), ifid END

lars@gateway:~$ influx -database ntopng -execute "show retention policies" name duration shardGroupDuration replicaN default

autogen 8760h0m0s 168h0m0s 1 true 1h 8760h0m0s 168h0m0s 1 false 1d 8760h0m0s 168h0m0s 1 false

Other charts also do not work: image

image

emanuele-f commented 5 years ago

Can you upgrade InfluxDB to the latest version (1.7.6) from https://portal.influxdata.com/downloads ?

dive76 commented 5 years ago

Have updated to 1.7.6 Now all works again!

emanuele-f commented 5 years ago

Great. I don't know if you run into a bugged version or not, but reading again the docs at https://docs.influxdata.com/influxdb/v1.7/query_language/data_exploration/#from-clause I have made a change to also specify the database in the fully qualified name in order to comply with the specs for the fully qualified name ebc5251a7ff2eab298fe0aa91c2ec2496d25e6d8.