search

netdata / netdata

Architected for speed. Automated for easy. Monitoring and troubleshooting, transformed!
https://www.netdata.cloud
GNU General Public License v3.0
72.01k stars 5.93k forks source link

On one host SOME prometheus metrics is inaccurate. #3195

Closed k0ste closed 6 years ago

k0ste commented 6 years ago

netdata version: 1.9.0 Release Prometheus query: http://192.168.100.15:19999/api/v1/allmetrics?format=prometheus&source=average netdata.conf:

## netdata.conf
# Ansible managed: /home/k0ste/ansible/roles/netdata/templates/netdata.j2 modified on 2017-04-13 12:27:57 by k0ste on WorkStation
# Do not edit manually

[global]
  memory deduplication (ksm) = no
  memory mode = save
  web files owner = root
  error log = stderr
  web files group = netdata
  access log = none
[plugins]
  fping = no
  node.d = no
  checks = no
  tc = no
[health]
  enabled = no
[backend]
  enabled = no
[registry]
  enabled = yes

kernel: 3.10.0-693.11.1.el7 (CentOS Linux release 7.4.1708 (Core)) Hardware: Dell PowerEdge R530 HDD (sd[a-h]): Toshiba X300 4-6Tb nvme2n1: Plextor M8PEY 1Tb

Look at this:

selection_003 selection_002

And at same time on netdata dashboard:

selection_001

Couple of drives have bullshit metrics. Error? Run fio on not rotational drive: selection_004 selection_005

And series looks like this:

netdata_disk_util___of_time_working_average{chart="disk_util.nvme2n1",family="nvme2n1",dimension="utilization"} 4.7565070 1514652852000

It seems to me that somewhere is set divisor.

netdata startup log ```shell 2017-12-30 23:52:38: netdata INFO : Adjusted my Out-Of-Memory (OOM) score from 0 to 1000. 2017-12-30 23:52:38: netdata INFO : netdata started on pid 215243. 2017-12-30 23:52:38: netdata ERROR: Registry: cannot open registry file: '/var/lib/netdata/registry/registry.db' (errno 2, No such file or directory) 2017-12-30 23:52:38: netdata INFO : Host 'ceph-osd5' (at registry as 'ceph-osd5') with guid 'da1db16c-41c4-11e7-bc9b-3cfdfea55f28' initialized, os 'linux', timezone 'Asia/Novosibirsk', tags '', update every 1, memory mode save, history entries 3996, streaming disabled (to '' with api key ''), health disabled, cache_dir '/var/cache/netdata', varlib_dir '/var/lib/netdata', health_log '/var/lib/netdata/health/health-log.db', alarms default handler '/usr/libexec/netdata/plugins.d/alarm-notify.sh', alarms default recipient 'root' 2017-12-30 23:52:38: netdata INFO : PROC Plugin thread created with task id 215245 2017-12-30 23:52:38: netdata INFO : IDLEJITTER thread created with task id 215248 2017-12-30 23:52:38: netdata INFO : CGROUP plugin thread created with task id 215247 2017-12-30 23:52:38: netdata INFO : DISKSPACE thread created with task id 215246 2017-12-30 23:52:38: netdata INFO : BACKEND: thread created with task id 215249 2017-12-30 23:52:38: netdata INFO : HEALTH thread created with task id 215250 2017-12-30 23:52:38: netdata INFO : Multi-threaded WEB SERVER thread created with task id 215252 2017-12-30 23:52:38: netdata INFO : PLUGINS.D thread created with task id 215251 2017-12-30 23:52:38: netdata INFO : BACKEND: thread exiting 2017-12-30 23:52:38: netdata INFO : netdata initialization completed. Enjoy real-time performance monitoring! 2017-12-30 23:52:38: netdata INFO : Listening on 'tcp:0.0.0.0:19999' 2017-12-30 23:52:38: netdata INFO : Listening on 'tcp:[::]:19999' 2017-12-30 23:52:38: netdata INFO : STATSD main thread created with task id 215253 2017-12-30 23:52:38: netdata INFO : PLUGINSD: '/usr/libexec/netdata/plugins.d/apps.plugin' running on pid 215255 2017-12-30 23:52:38: netdata INFO : PLUGINSD: '/usr/libexec/netdata/plugins.d/charts.d.plugin' running on pid 215257 2017-12-30 23:52:38: netdata INFO : PLUGINSD: '/usr/libexec/netdata/plugins.d/python.d.plugin' running on pid 215259 2017-12-30 23:52:38: netdata ERROR: PLUGINSD: Cannot open plugins directory '/etc/netdata/custom-plugins.d'. (errno 2, No such file or directory) 2017-12-30 23:52:38: netdata ERROR: LISTENER: IPv6 bind() on ip '::1' port 8125, socktype 2 failed. (errno 99, Cannot assign requested address) 2017-12-30 23:52:38: netdata ERROR: LISTENER: Cannot bind to ip '::1', port 8125 2017-12-30 23:52:38: netdata ERROR: LISTENER: IPv6 bind() on ip '::1' port 8125, socktype 1 failed. (errno 99, Cannot assign requested address) 2017-12-30 23:52:38: netdata ERROR: LISTENER: Cannot bind to ip '::1', port 8125 2017-12-30 23:52:38: netdata INFO : LISTENER: Listen socket udp:127.0.0.1:8125 opened successfully. 2017-12-30 23:52:38: netdata INFO : LISTENER: Listen socket tcp:127.0.0.1:8125 opened successfully. 2017-12-30 23:52:38: netdata INFO : STATSD collector thread No 2 created with task id 215260 2017-12-30 23:52:38: netdata INFO : POLLFD: LISTENER: listening on 'udp:127.0.0.1:8125' 2017-12-30 23:52:38: netdata INFO : POLLFD: LISTENER: listening on 'tcp:127.0.0.1:8125' 2017-12-30 23:52:38: apps.plugin INFO : started on pid 215255 2017-12-30 23:52:38: charts.d: INFO: main: started from '/usr/libexec/netdata/plugins.d/charts.d.plugin' with options: 1 2017-12-30 23:52:38: charts.d: INFO: apache: is disabled. Add a line with apache=force in /etc/netdata/charts.d.conf to enable it (or remove the line that disables it). 2017-12-30 23:52:38: charts.d: INFO: cpu_apps: is disabled. Add a line with cpu_apps=force in /etc/netdata/charts.d.conf to enable it (or remove the line that disables it). 2017-12-30 23:52:38: charts.d: INFO: cpufreq: is disabled. Add a line with cpufreq=force in /etc/netdata/charts.d.conf to enable it (or remove the line that disables it). 2017-12-30 23:52:38: charts.d: INFO: example: is disabled. Add a line with example=force in /etc/netdata/charts.d.conf to enable it (or remove the line that disables it). 2017-12-30 23:52:38: charts.d: INFO: exim: is disabled. Add a line with exim=force in /etc/netdata/charts.d.conf to enable it (or remove the line that disables it). 2017-12-30 23:52:38: charts.d: INFO: hddtemp: is disabled. Add a line with hddtemp=force in /etc/netdata/charts.d.conf to enable it (or remove the line that disables it). 2017-12-30 23:52:38: charts.d: INFO: load_average: is disabled. Add a line with load_average=force in /etc/netdata/charts.d.conf to enable it (or remove the line that disables it). 2017-12-30 23:52:38: charts.d: INFO: mem_apps: is disabled. Add a line with mem_apps=force in /etc/netdata/charts.d.conf to enable it (or remove the line that disables it). 2017-12-30 23:52:38: charts.d: INFO: mysql: is disabled. Add a line with mysql=force in /etc/netdata/charts.d.conf to enable it (or remove the line that disables it). 2017-12-30 23:52:38: charts.d: INFO: nginx: is disabled. Add a line with nginx=force in /etc/netdata/charts.d.conf to enable it (or remove the line that disables it). 2017-12-30 23:52:38: charts.d: INFO: phpfpm: is disabled. Add a line with phpfpm=force in /etc/netdata/charts.d.conf to enable it (or remove the line that disables it). 2017-12-30 23:52:38: charts.d: INFO: postfix: is disabled. Add a line with postfix=force in /etc/netdata/charts.d.conf to enable it (or remove the line that disables it). 2017-12-30 23:52:38: charts.d: INFO: sensors: is disabled. Add a line with sensors=force in /etc/netdata/charts.d.conf to enable it (or remove the line that disables it). 2017-12-30 23:52:38: charts.d: INFO: squid: is disabled. Add a line with squid=force in /etc/netdata/charts.d.conf to enable it (or remove the line that disables it). 2017-12-30 23:52:38: charts.d: INFO: tomcat: is disabled. Add a line with tomcat=force in /etc/netdata/charts.d.conf to enable it (or remove the line that disables it). 2017-12-30 23:52:39: charts.d: WARNING: ap: command 'iw' is not found in /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin. 2017-12-30 23:52:39: charts.d: ERROR: ap: module's 'ap' check() function reports failure. 2017-12-30 23:52:39: charts.d: WARNING: apcupsd: command 'apcaccess' is not found in /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin. 2017-12-30 23:52:39: charts.d: ERROR: apcupsd: module's 'apcupsd' check() function reports failure. 2017-12-30 23:52:39: charts.d: WARNING: nut: command 'upsc' is not found in /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin. 2017-12-30 23:52:39: charts.d: ERROR: nut: module's 'nut' check() function reports failure. 2017-12-30 23:52:39: python.d INFO: plugin: main: Using python 2 2017-12-30 23:52:39: charts.d: WARNING: opensips: command 'opensipsctl' is not found in /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin. 2017-12-30 23:52:39: charts.d: ERROR: opensips: module's 'opensips' check() function reports failure. 2017-12-30 23:52:39: charts.d: FATAL: main: No charts to collect data from. 2017-12-30 23:52:39: netdata INFO : PLUGINSD: '/usr/libexec/netdata/plugins.d/charts.d.plugin' called DISABLE. Disabling it. 2017-12-30 23:52:39: netdata ERROR: PLUGINSD: plugin '/usr/libexec/netdata/plugins.d/charts.d.plugin' disconnected. 2017-12-30 23:52:39: netdata INFO : PLUGINSD: '/usr/libexec/netdata/plugins.d/charts.d.plugin' on pid 215257 stopped after 0 successful data collections (ENDs). 2017-12-30 23:52:39: netdata ERROR: PLUGINSD: '/usr/libexec/netdata/plugins.d/charts.d.plugin' (pid 215257) does not generate useful output but it reports success (exits with 0). Will not start it again - it is disabled.. (errno 9, Bad file descriptor) 2017-12-30 23:52:39: python.d ERROR: plugin: main: module load config: 'cpuidle' => [FAILED] 2017-12-30 23:52:39: python.d ERROR: plugin: main: load config error : [Errno 2] No such file or directory: '/etc/netdata/python.d/cpuidle.conf' 2017-12-30 23:52:39: python.d ERROR: apache: localhost: Url: http://localhost/server-status?auto. Error: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /server-status?auto (Caused by ProtocolError('Connection aborted.', error(113, 'No route to host'))) 2017-12-30 23:52:39: python.d ERROR: apache: localhost: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: apache: localipv4: Url: http://127.0.0.1/server-status?auto. Error: HTTPConnectionPool(host='127.0.0.1', port=80): Max retries exceeded with url: /server-status?auto (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused'))) 2017-12-30 23:52:39: python.d ERROR: apache: localipv4: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: apache: localipv6: Url: http://::1/server-status?auto. Error: Failed to parse: ::1 2017-12-30 23:52:39: python.d ERROR: apache: localipv6: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: beanstalk: beanstalk: 'beanstalkc' module is needed to use beanstalk.chart.py 2017-12-30 23:52:39: python.d ERROR: beanstalk: beanstalk: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: bind_rndc: bind_rndc: Can't locate "rndc" binary or binary is not executable by netdata 2017-12-30 23:52:39: python.d ERROR: bind_rndc: bind_rndc: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: couchdb: localhost: Url: http://127.0.0.1:5984/_node/couchdb@127.0.0.1/_stats. Error: HTTPConnectionPool(host='127.0.0.1', port=5984): Max retries exceeded with url: /_node/couchdb@127.0.0.1/_stats (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused'))) 2017-12-30 23:52:39: python.d ERROR: couchdb: localhost: Url: http://127.0.0.1:5984/_active_tasks. Error: HTTPConnectionPool(host='127.0.0.1', port=5984): Max retries exceeded with url: /_active_tasks (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused'))) 2017-12-30 23:52:39: python.d ERROR: couchdb: localhost: Url: http://127.0.0.1:5984/_node/couchdb@127.0.0.1/_system. Error: HTTPConnectionPool(host='127.0.0.1', port=5984): Max retries exceeded with url: /_node/couchdb@127.0.0.1/_system (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused'))) 2017-12-30 23:52:39: python.d ERROR: couchdb: localhost: _get_data() returned no data or type is not 2017-12-30 23:52:39: python.d ERROR: couchdb: localhost: check() => [FAILED] 2017-12-30 23:52:39: python.d INFO: cpufreq: cpufreq: check() => [OK] 2017-12-30 23:52:39: python.d ERROR: cpuidle: cpuidle: couldn't find cstate stats 2017-12-30 23:52:39: python.d ERROR: cpuidle: cpuidle: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: dns_query_time: dns_query_time: 'python-dnspython' package is needed to use dns_query_time.chart.py 2017-12-30 23:52:39: python.d ERROR: dns_query_time: dns_query_time: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: dnsdist: dnsdist: URL is not defined or type is not 2017-12-30 23:52:39: python.d ERROR: dnsdist: dnsdist: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: dovecot: localhost: Failed to connect to "127.0.0.1", port 24242, error: [Errno 111] Connection refused 2017-12-30 23:52:39: python.d ERROR: dovecot: localhost: Failed to connect to "::1", port 24242, error: [Errno 113] No route to host 2017-12-30 23:52:39: python.d ERROR: dovecot: localhost: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: dovecot: localipv4: Failed to connect to "127.0.0.1", port 24242, error: [Errno 111] Connection refused 2017-12-30 23:52:39: python.d ERROR: dovecot: localipv4: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: dovecot: localipv6: Failed to connect to "::1", port 24242, error: [Errno 113] No route to host 2017-12-30 23:52:39: python.d ERROR: dovecot: localipv6: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: dovecot: localsocket: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: elasticsearch: local: Url: http://127.0.0.1:9200/_cluster/health. Error: HTTPConnectionPool(host='127.0.0.1', port=9200): Max retries exceeded with url: /_cluster/health (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused'))) 2017-12-30 23:52:39: python.d ERROR: elasticsearch: local: Url: http://127.0.0.1:9200/_nodes/_local/stats. Error: HTTPConnectionPool(host='127.0.0.1', port=9200): Max retries exceeded with url: /_nodes/_local/stats (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused'))) 2017-12-30 23:52:39: python.d ERROR: elasticsearch: local: Url: http://127.0.0.1:9200/_cluster/stats. Error: HTTPConnectionPool(host='127.0.0.1', port=9200): Max retries exceeded with url: /_cluster/stats (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused'))) 2017-12-30 23:52:39: python.d ERROR: elasticsearch: local: _get_data() returned no data or type is not 2017-12-30 23:52:39: python.d ERROR: elasticsearch: local: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: exim: local: Can't locate "exim -bpc" binary 2017-12-30 23:52:39: python.d ERROR: exim: local: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: fail2ban: fail2ban: /var/log/fail2ban.log is not readable or empty 2017-12-30 23:52:39: python.d ERROR: fail2ban: fail2ban: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: freeradius: freeradius: Can't locate "radclient" binary or binary is not executable by netdata 2017-12-30 23:52:39: python.d ERROR: freeradius: freeradius: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: haproxy: haproxy: URL is not defined or type is not 2017-12-30 23:52:39: python.d ERROR: haproxy: haproxy: check() => [FAILED] 2017-12-30 23:52:39: python.d INFO: hddtemp: localhost: Autodetecting disks 2017-12-30 23:52:39: python.d ERROR: hddtemp: localhost: Failed to connect to "127.0.0.1", port 7634, error: [Errno 111] Connection refused 2017-12-30 23:52:39: python.d ERROR: hddtemp: localhost: Failed to connect to "::1", port 7634, error: [Errno 113] No route to host 2017-12-30 23:52:39: python.d ERROR: hddtemp: localhost: no data received 2017-12-30 23:52:39: python.d ERROR: hddtemp: localhost: check() => [FAILED] 2017-12-30 23:52:39: python.d INFO: hddtemp: localipv4: Autodetecting disks 2017-12-30 23:52:39: python.d ERROR: hddtemp: localipv4: Failed to connect to "127.0.0.1", port 7634, error: [Errno 111] Connection refused 2017-12-30 23:52:39: python.d ERROR: hddtemp: localipv4: no data received 2017-12-30 23:52:39: python.d ERROR: hddtemp: localipv4: check() => [FAILED] 2017-12-30 23:52:39: python.d INFO: hddtemp: localipv6: Autodetecting disks 2017-12-30 23:52:39: python.d ERROR: hddtemp: localipv6: Failed to connect to "::1", port 7634, error: [Errno 113] No route to host 2017-12-30 23:52:39: python.d ERROR: hddtemp: localipv6: no data received 2017-12-30 23:52:39: python.d ERROR: hddtemp: localipv6: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: ipfs: localhost: Url: http://localhost:5001/api/v0/pin/ls. Error: HTTPConnectionPool(host='localhost', port=5001): Max retries exceeded with url: /api/v0/pin/ls (Caused by ProtocolError('Connection aborted.', error(113, 'No route to host'))) 2017-12-30 23:52:39: python.d ERROR: ipfs: localhost: Url: http://localhost:5001/api/v0/stats/repo. Error: HTTPConnectionPool(host='localhost', port=5001): Max retries exceeded with url: /api/v0/stats/repo (Caused by ProtocolError('Connection aborted.', error(113, 'No route to host'))) 2017-12-30 23:52:39: python.d ERROR: ipfs: localhost: Url: http://localhost:5001/api/v0/stats/bw. Error: HTTPConnectionPool(host='localhost', port=5001): Max retries exceeded with url: /api/v0/stats/bw (Caused by ProtocolError('Connection aborted.', error(113, 'No route to host'))) 2017-12-30 23:52:39: python.d ERROR: ipfs: localhost: Url: http://localhost:5001/api/v0/config/show. Error: HTTPConnectionPool(host='localhost', port=5001): Max retries exceeded with url: /api/v0/config/show (Caused by ProtocolError('Connection aborted.', error(113, 'No route to host'))) 2017-12-30 23:52:39: python.d ERROR: ipfs: localhost: Url: http://localhost:5001/api/v0/swarm/peers. Error: HTTPConnectionPool(host='localhost', port=5001): Max retries exceeded with url: /api/v0/swarm/peers (Caused by ProtocolError('Connection aborted.', error(113, 'No route to host'))) 2017-12-30 23:52:39: python.d ERROR: ipfs: localhost: _get_data() returned no data or type is not 2017-12-30 23:52:39: python.d ERROR: ipfs: localhost: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: isc_dhcpd: isc_dhcpd: 'python-ipaddress' module is needed 2017-12-30 23:52:39: python.d ERROR: isc_dhcpd: isc_dhcpd: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mdstat: mdstat: Failed to read data from /proc/mdstat or there is no active arrays 2017-12-30 23:52:39: python.d ERROR: mdstat: mdstat: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: memcached: localhost: Failed to connect to "127.0.0.1", port 11211, error: [Errno 111] Connection refused 2017-12-30 23:52:39: python.d ERROR: memcached: localhost: Failed to connect to "::1", port 11211, error: [Errno 113] No route to host 2017-12-30 23:52:39: python.d ERROR: memcached: localhost: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: memcached: localipv4: Failed to connect to "127.0.0.1", port 11211, error: [Errno 111] Connection refused 2017-12-30 23:52:39: python.d ERROR: memcached: localipv4: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: memcached: localipv6: Failed to connect to "::1", port 11211, error: [Errno 113] No route to host 2017-12-30 23:52:39: python.d ERROR: memcached: localipv6: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mongodb: local: Pymongo module is needed to use mongodb.chart.py 2017-12-30 23:52:39: python.d ERROR: mongodb: local: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: mycnf1: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: mycnf1: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: mycnf2: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: mycnf2: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: debiancnf: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: debiancnf: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: socket1: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: socket1: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: socket2: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: socket2: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: socket3: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: socket3: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: socket4: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: socket4: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: tcp: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: tcp: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: tcpipv4: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: tcpipv4: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: tcpipv6: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: tcpipv6: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: mycnf1_root: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: mycnf1_root: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: mycnf2_root: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: mycnf2_root: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: socket1_root: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: socket1_root: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: socket2_root: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: socket2_root: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: socket3_root: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: socket3_root: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: socket4_root: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: socket4_root: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: tcp_root: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: tcp_root: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: tcpipv4_root: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: tcpipv4_root: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: tcpipv6_root: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: tcpipv6_root: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: mycnf1_netdata: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: mycnf1_netdata: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: mycnf2_netdata: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: mycnf2_netdata: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: socket1_netdata: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: socket1_netdata: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: socket2_netdata: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: socket2_netdata: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: socket3_netdata: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: socket3_netdata: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: socket4_netdata: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: socket4_netdata: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: tcp_netdata: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: tcp_netdata: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: tcpipv4_netdata: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: tcpipv4_netdata: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: mysql: tcpipv6_netdata: MySQLdb or PyMySQL module is needed to use mysql.chart.py plugin 2017-12-30 23:52:39: python.d ERROR: mysql: tcpipv6_netdata: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: nginx: localhost: Url: http://localhost/stub_status. Error: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /stub_status (Caused by ProtocolError('Connection aborted.', error(113, 'No route to host'))) 2017-12-30 23:52:39: python.d ERROR: nginx: localhost: _get_data() returned no data or type is not 2017-12-30 23:52:39: python.d ERROR: nginx: localhost: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: nginx: localipv4: Url: http://127.0.0.1/stub_status. Error: HTTPConnectionPool(host='127.0.0.1', port=80): Max retries exceeded with url: /stub_status (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused'))) 2017-12-30 23:52:39: python.d ERROR: nginx: localipv4: _get_data() returned no data or type is not 2017-12-30 23:52:39: python.d ERROR: nginx: localipv4: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: nginx: localipv6: Url: http://[::1]/stub_status. Error: HTTPConnectionPool(host='::1', port=80): Max retries exceeded with url: /stub_status (Caused by ProtocolError('Connection aborted.', error(113, 'No route to host'))) 2017-12-30 23:52:39: python.d ERROR: nginx: localipv6: _get_data() returned no data or type is not 2017-12-30 23:52:39: python.d ERROR: nginx: localipv6: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: nsd: local: Can't locate "nsd-control stats_noreset" binary 2017-12-30 23:52:39: python.d ERROR: nsd: local: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: ovpn_status_log: ovpn_status_log: 'log_path' is not defined 2017-12-30 23:52:39: python.d ERROR: ovpn_status_log: ovpn_status_log: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: phpfpm: localhost: Url: http://localhost/status?full&json. Error: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /status?full&json (Caused by ProtocolError('Connection aborted.', error(113, 'No route to host'))) 2017-12-30 23:52:39: python.d ERROR: phpfpm: localhost: _get_data() returned no data or type is not 2017-12-30 23:52:39: python.d ERROR: phpfpm: localhost: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: phpfpm: localipv4: Url: http://127.0.0.1/status?full&json. Error: HTTPConnectionPool(host='127.0.0.1', port=80): Max retries exceeded with url: /status?full&json (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused'))) 2017-12-30 23:52:39: python.d ERROR: phpfpm: localipv4: _get_data() returned no data or type is not 2017-12-30 23:52:39: python.d ERROR: phpfpm: localipv4: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: phpfpm: localipv6: Url: http://::1/status?full&json. Error: Failed to parse: ::1 2017-12-30 23:52:39: python.d ERROR: phpfpm: localipv6: _get_data() returned no data or type is not 2017-12-30 23:52:39: python.d ERROR: phpfpm: localipv6: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: postfix: local: Can't locate "postqueue -p" binary 2017-12-30 23:52:39: python.d ERROR: postfix: local: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: postgres: socket: 'python-psycopg2' module is needed to use postgres.chart.py 2017-12-30 23:52:39: python.d ERROR: postgres: socket: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: postgres: tcp: 'python-psycopg2' module is needed to use postgres.chart.py 2017-12-30 23:52:39: python.d ERROR: postgres: tcp: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: postgres: tcpipv4: 'python-psycopg2' module is needed to use postgres.chart.py 2017-12-30 23:52:39: python.d ERROR: postgres: tcpipv4: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: postgres: tcpipv6: 'python-psycopg2' module is needed to use postgres.chart.py 2017-12-30 23:52:39: python.d ERROR: postgres: tcpipv6: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: powerdns: powerdns: URL is not defined or type is not 2017-12-30 23:52:39: python.d ERROR: powerdns: powerdns: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: rabbitmq: local: Url: http://127.0.0.1:15672/api/nodes. Error: HTTPConnectionPool(host='127.0.0.1', port=15672): Max retries exceeded with url: /api/nodes (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused'))) 2017-12-30 23:52:39: python.d ERROR: rabbitmq: local: Url: http://127.0.0.1:15672/api/overview. Error: HTTPConnectionPool(host='127.0.0.1', port=15672): Max retries exceeded with url: /api/overview (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused'))) 2017-12-30 23:52:39: python.d ERROR: rabbitmq: local: _get_data() returned no data or type is not 2017-12-30 23:52:39: python.d ERROR: rabbitmq: local: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: redis: socket1: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: redis: socket2: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: redis: socket3: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: redis: localhost: Failed to connect to "127.0.0.1", port 6379, error: [Errno 111] Connection refused 2017-12-30 23:52:39: python.d ERROR: redis: localhost: Failed to connect to "::1", port 6379, error: [Errno 113] No route to host 2017-12-30 23:52:39: python.d ERROR: redis: localhost: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: redis: localipv4: Failed to connect to "127.0.0.1", port 6379, error: [Errno 111] Connection refused 2017-12-30 23:52:39: python.d ERROR: redis: localipv4: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: redis: localipv6: Failed to connect to "::1", port 6379, error: [Errno 113] No route to host 2017-12-30 23:52:39: python.d ERROR: redis: localipv6: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: retroshare: localhost: Url: http://localhost:9090/api/v2/stats. Error: HTTPConnectionPool(host='localhost', port=9090): Max retries exceeded with url: /api/v2/stats (Caused by ProtocolError('Connection aborted.', error(113, 'No route to host'))) 2017-12-30 23:52:39: python.d ERROR: retroshare: localhost: _get_data() returned no data or type is not 2017-12-30 23:52:39: python.d ERROR: retroshare: localhost: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: samba: samba: Can't locate 'sudo' or 'smbstatus' binary 2017-12-30 23:52:39: python.d ERROR: samba: samba: check() => [FAILED] 2017-12-30 23:52:39: python.d INFO: sensors: sensors: check() => [OK] 2017-12-30 23:52:39: python.d INFO: smartd_log: smartd_log: check() => [OK] 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128old: Failed to connect to "127.0.0.1", port 3128, error: [Errno 111] Connection refused 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128old: Failed to connect to "::1", port 3128, error: [Errno 113] No route to host 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128old: invalid data received 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128old: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080old: Failed to connect to "127.0.0.1", port 8080, error: [Errno 111] Connection refused 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080old: Failed to connect to "::1", port 8080, error: [Errno 113] No route to host 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080old: invalid data received 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080old: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128new: Failed to connect to "127.0.0.1", port 3128, error: [Errno 111] Connection refused 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128new: Failed to connect to "::1", port 3128, error: [Errno 113] No route to host 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128new: invalid data received 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128new: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080new: Failed to connect to "127.0.0.1", port 8080, error: [Errno 111] Connection refused 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080new: Failed to connect to "::1", port 8080, error: [Errno 113] No route to host 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080new: invalid data received 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080new: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128oldipv4: Failed to connect to "127.0.0.1", port 3128, error: [Errno 111] Connection refused 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128oldipv4: invalid data received 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128oldipv4: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080oldipv4: Failed to connect to "127.0.0.1", port 8080, error: [Errno 111] Connection refused 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080oldipv4: invalid data received 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080oldipv4: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128newipv4: Failed to connect to "127.0.0.1", port 3128, error: [Errno 111] Connection refused 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128newipv4: invalid data received 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128newipv4: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080newipv4: Failed to connect to "127.0.0.1", port 8080, error: [Errno 111] Connection refused 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080newipv4: invalid data received 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080newipv4: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128oldipv6: Failed to connect to "::1", port 3128, error: [Errno 113] No route to host 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128oldipv6: invalid data received 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128oldipv6: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080oldipv6: Failed to connect to "::1", port 8080, error: [Errno 113] No route to host 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080oldipv6: invalid data received 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080oldipv6: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128newipv6: Failed to connect to "::1", port 3128, error: [Errno 113] No route to host 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128newipv6: invalid data received 2017-12-30 23:52:39: python.d ERROR: squid: tcp3128newipv6: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080newipv6: Failed to connect to "::1", port 8080, error: [Errno 113] No route to host 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080newipv6: invalid data received 2017-12-30 23:52:39: python.d ERROR: squid: tcp8080newipv6: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: tomcat: localhost: Url: http://localhost:8080/manager/status?XML=true. Error: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /manager/status?XML=true (Caused by ProtocolError('Connection aborted.', error(113, 'No route to host'))) 2017-12-30 23:52:39: python.d ERROR: tomcat: localhost: _get_data() returned no data or type is not 2017-12-30 23:52:39: python.d ERROR: tomcat: localhost: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: tomcat: localipv4: Url: http://127.0.0.1:8080/manager/status?XML=true. Error: HTTPConnectionPool(host='127.0.0.1', port=8080): Max retries exceeded with url: /manager/status?XML=true (Caused by ProtocolError('Connection aborted.', error(111, 'Connection refused'))) 2017-12-30 23:52:39: python.d ERROR: tomcat: localipv4: _get_data() returned no data or type is not 2017-12-30 23:52:39: python.d ERROR: tomcat: localipv4: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: tomcat: localipv6: Url: http://[::1]:8080/manager/status?XML=true. Error: HTTPConnectionPool(host='::1', port=8080): Max retries exceeded with url: /manager/status?XML=true (Caused by ProtocolError('Connection aborted.', error(113, 'No route to host'))) 2017-12-30 23:52:39: python.d ERROR: tomcat: localipv6: _get_data() returned no data or type is not 2017-12-30 23:52:39: python.d ERROR: tomcat: localipv6: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: varnish: varnish: Can't locate 'varnishstat' binary or binary is not executable by user netdata 2017-12-30 23:52:39: python.d ERROR: varnish: varnish: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: web_log: nginx_log: /var/log/nginx/access.log not readable or not exist 2017-12-30 23:52:39: python.d ERROR: web_log: nginx_log: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: web_log: nginx_log2: /var/log/nginx/localhost.access_log not readable or not exist 2017-12-30 23:52:39: python.d ERROR: web_log: nginx_log2: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: web_log: apache_log: /var/log/apache2/access.log not readable or not exist 2017-12-30 23:52:39: python.d ERROR: web_log: apache_log: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: web_log: apache_log2: /var/log/apache2/access_log not readable or not exist 2017-12-30 23:52:39: python.d ERROR: web_log: apache_log2: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: web_log: apache_log3: /var/log/httpd/access_log not readable or not exist 2017-12-30 23:52:39: python.d ERROR: web_log: apache_log3: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: web_log: apache_vhosts_log: /var/log/apache2/other_vhosts_access.log not readable or not exist 2017-12-30 23:52:39: python.d ERROR: web_log: apache_vhosts_log: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: web_log: gunicorn_log: /var/log/gunicorn/access.log not readable or not exist 2017-12-30 23:52:39: python.d ERROR: web_log: gunicorn_log: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: web_log: gunicorn_log2: /var/log/gunicorn/gunicorn-access.log not readable or not exist 2017-12-30 23:52:39: python.d ERROR: web_log: gunicorn_log2: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: web_log: apache_cache: /var/log/apache/cache.log not readable or not exist 2017-12-30 23:52:39: python.d ERROR: web_log: apache_cache: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: web_log: apache2_cache: /var/log/apache2/cache.log not readable or not exist 2017-12-30 23:52:39: python.d ERROR: web_log: apache2_cache: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: web_log: httpd_cache: /var/log/httpd/cache.log not readable or not exist 2017-12-30 23:52:39: python.d ERROR: web_log: httpd_cache: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: web_log: squid_log1: /var/log/squid3/access.log not readable or not exist 2017-12-30 23:52:39: python.d ERROR: web_log: squid_log1: check() => [FAILED] 2017-12-30 23:52:39: python.d ERROR: web_log: squid_log2: /var/log/squid/access.log not readable or not exist 2017-12-30 23:52:39: python.d ERROR: web_log: squid_log2: check() => [FAILED] 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.cpu' on host 'ceph-osd5' is 0.42480 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.cpu_user' on host 'ceph-osd5' is 0.42477 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.cpu_system' on host 'ceph-osd5' is 0.42476 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.threads' on host 'ceph-osd5' is 0.42475 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.processes' on host 'ceph-osd5' is 0.42473 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.mem' on host 'ceph-osd5' is 0.42472 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.vmem' on host 'ceph-osd5' is 0.42471 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.minor_faults' on host 'ceph-osd5' is 0.42470 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.major_faults' on host 'ceph-osd5' is 0.42469 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.lreads' on host 'ceph-osd5' is 0.42468 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.lwrites' on host 'ceph-osd5' is 0.42466 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.preads' on host 'ceph-osd5' is 0.42465 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.pwrites' on host 'ceph-osd5' is 0.42464 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.files' on host 'ceph-osd5' is 0.42463 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.sockets' on host 'ceph-osd5' is 0.42462 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.pipes' on host 'ceph-osd5' is 0.42461 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.cpu' on host 'ceph-osd5' is 0.42459 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.cpu_user' on host 'ceph-osd5' is 0.42458 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.cpu_system' on host 'ceph-osd5' is 0.42457 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.threads' on host 'ceph-osd5' is 0.42456 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.processes' on host 'ceph-osd5' is 0.42455 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.mem' on host 'ceph-osd5' is 0.42454 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.vmem' on host 'ceph-osd5' is 0.42453 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.minor_faults' on host 'ceph-osd5' is 0.42452 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.major_faults' on host 'ceph-osd5' is 0.42452 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.lreads' on host 'ceph-osd5' is 0.42451 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.lwrites' on host 'ceph-osd5' is 0.42450 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.preads' on host 'ceph-osd5' is 0.42449 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.pwrites' on host 'ceph-osd5' is 0.42448 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.files' on host 'ceph-osd5' is 0.42447 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.sockets' on host 'ceph-osd5' is 0.42446 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.pipes' on host 'ceph-osd5' is 0.42446 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.cpu' on host 'ceph-osd5' is 0.42445 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.cpu_user' on host 'ceph-osd5' is 0.42444 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.cpu_system' on host 'ceph-osd5' is 0.42443 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.threads' on host 'ceph-osd5' is 0.42442 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.processes' on host 'ceph-osd5' is 0.42441 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.mem' on host 'ceph-osd5' is 0.42441 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.vmem' on host 'ceph-osd5' is 0.42440 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.minor_faults' on host 'ceph-osd5' is 0.42439 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.major_faults' on host 'ceph-osd5' is 0.42438 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.lreads' on host 'ceph-osd5' is 0.42437 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.lwrites' on host 'ceph-osd5' is 0.42436 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.preads' on host 'ceph-osd5' is 0.42435 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.pwrites' on host 'ceph-osd5' is 0.42435 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.files' on host 'ceph-osd5' is 0.42434 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.sockets' on host 'ceph-osd5' is 0.42433 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.pipes' on host 'ceph-osd5' is 0.42432 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.cpu' on host 'ceph-osd5' is 0.42430 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.cpu_user' on host 'ceph-osd5' is 0.42429 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.cpu_system' on host 'ceph-osd5' is 0.42428 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.threads' on host 'ceph-osd5' is 0.42427 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.processes' on host 'ceph-osd5' is 0.42425 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.mem' on host 'ceph-osd5' is 0.42424 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.vmem' on host 'ceph-osd5' is 0.42423 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.minor_faults' on host 'ceph-osd5' is 0.42422 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.major_faults' on host 'ceph-osd5' is 0.42421 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.lreads' on host 'ceph-osd5' is 0.42420 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.lwrites' on host 'ceph-osd5' is 0.42418 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.preads' on host 'ceph-osd5' is 0.42417 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.pwrites' on host 'ceph-osd5' is 0.42416 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.files' on host 'ceph-osd5' is 0.42414 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.sockets' on host 'ceph-osd5' is 0.42413 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'apps.pipes' on host 'ceph-osd5' is 0.42412 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.cpu' on host 'ceph-osd5' is 0.42411 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.cpu_user' on host 'ceph-osd5' is 0.42410 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.cpu_system' on host 'ceph-osd5' is 0.42409 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.threads' on host 'ceph-osd5' is 0.42409 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.processes' on host 'ceph-osd5' is 0.42408 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.mem' on host 'ceph-osd5' is 0.42407 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.vmem' on host 'ceph-osd5' is 0.42406 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.minor_faults' on host 'ceph-osd5' is 0.42405 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.major_faults' on host 'ceph-osd5' is 0.42404 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.lreads' on host 'ceph-osd5' is 0.42403 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.lwrites' on host 'ceph-osd5' is 0.42402 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.preads' on host 'ceph-osd5' is 0.42402 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.pwrites' on host 'ceph-osd5' is 0.42401 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.files' on host 'ceph-osd5' is 0.42400 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.sockets' on host 'ceph-osd5' is 0.42399 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'users.pipes' on host 'ceph-osd5' is 0.42398 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.cpu' on host 'ceph-osd5' is 0.42397 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.cpu_user' on host 'ceph-osd5' is 0.42397 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.cpu_system' on host 'ceph-osd5' is 0.42396 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.threads' on host 'ceph-osd5' is 0.42395 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.processes' on host 'ceph-osd5' is 0.42394 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.mem' on host 'ceph-osd5' is 0.42393 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.vmem' on host 'ceph-osd5' is 0.42392 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.minor_faults' on host 'ceph-osd5' is 0.42392 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.major_faults' on host 'ceph-osd5' is 0.42391 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.lreads' on host 'ceph-osd5' is 0.42390 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.lwrites' on host 'ceph-osd5' is 0.42389 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.preads' on host 'ceph-osd5' is 0.42388 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.pwrites' on host 'ceph-osd5' is 0.42387 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.files' on host 'ceph-osd5' is 0.42387 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.sockets' on host 'ceph-osd5' is 0.42386 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'groups.pipes' on host 'ceph-osd5' is 0.42385 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'smartd_log.attr_id_raw_1' on host 'ceph-osd5' is 0.29138 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'smartd_log.attr_id_normalized_197' on host 'ceph-osd5' is 0.29135 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'smartd_log.attr_id_normalized_198' on host 'ceph-osd5' is 0.29134 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'smartd_log.attr_id_raw_9' on host 'ceph-osd5' is 0.29132 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'netdata.runtime_smartd_log' on host 'ceph-osd5' is 0.29131 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'cpu.cpufreq' on host 'ceph-osd5' is 0.29130 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'netdata.runtime_cpufreq' on host 'ceph-osd5' is 0.29128 secs in the future. Adjusting it to current time. 2017-12-30 23:52:41: netdata INFO : RRD database for chart 'netdata.runtime_sensors' on host 'ceph-osd5' is 0.29126 secs in the future. Adjusting it to current time. 2017-12-30 23:52:47: netdata ERROR: PROCFILE: Cannot open file '/proc/net/ip_vs_stats' (errno 2, No such file or directory) 2017-12-30 23:52:47: netdata ERROR: PROCFILE: Cannot open file '/proc/net/stat/nf_conntrack' (errno 2, No such file or directory) 2017-12-30 23:52:47: netdata ERROR: PROCFILE: Cannot open file '/proc/net/stat/synproxy' (errno 2, No such file or directory) 2017-12-30 23:52:47: netdata ERROR: PROCFILE: Cannot open file '/proc/net/rpc/nfsd' (errno 2, No such file or directory) 2017-12-30 23:52:47: netdata ERROR: PROCFILE: Cannot open file '/proc/net/rpc/nfs' (errno 2, No such file or directory) 2017-12-30 23:52:47: netdata ERROR: PROCFILE: Cannot open file '/proc/spl/kstat/zfs/arcstats' (errno 2, No such file or directory) 2017-12-30 23:52:48: netdata ERROR: heartbeat missed 7999862 microseconds 2017-12-30 23:52:49: netdata INFO : PLUGINSD: '/usr/libexec/netdata/plugins.d/charts.d.plugin' thread exiting 2017-12-30 23:52:50: python.d ERROR: smartd_log: smartd_log: chart 'smartd_log.attr_id_raw_194' was suppressed due to non updating 2017-12-30 23:52:50: python.d ERROR: smartd_log: smartd_log: chart 'smartd_log.attr_id_normalized_200' was suppressed due to non updating 2017-12-30 23:52:50: python.d ERROR: smartd_log: smartd_log: chart 'smartd_log.attr_id_raw_200' was suppressed due to non updating 2017-12-30 23:55:28: netdata INFO : SIGNAL: Received SIGTERM. Cleaning up to exit... 2017-12-30 23:55:28: netdata INFO : Cleaning up database [1 hosts(s)]... 2017-12-30 23:55:28: netdata INFO : Cleaning up database of host 'ceph-osd5'... 2017-12-30 23:55:28: netdata ERROR: PLUGINSD: /usr/libexec/netdata/plugins.d/apps.plugin : read failed. 2017-12-30 23:55:28: netdata ERROR: PLUGINSD: plugin '/usr/libexec/netdata/plugins.d/apps.plugin' disconnected. 2017-12-30 23:55:28: netdata INFO : PLUGINSD: '/usr/libexec/netdata/plugins.d/apps.plugin' on pid 215255 stopped after 8840 successful data collections (ENDs). 2017-12-30 23:55:28: netdata ERROR: child pid 215255 killed by signal 15. (errno 9, Bad file descriptor) 2017-12-30 23:55:28: netdata INFO : PLUGINSD: '/usr/libexec/netdata/plugins.d/apps.plugin' thread exiting 2017-12-30 23:55:28: netdata ERROR: PLUGINSD: /usr/libexec/netdata/plugins.d/python.d.plugin : read failed. 2017-12-30 23:55:28: netdata ERROR: PLUGINSD: plugin '/usr/libexec/netdata/plugins.d/python.d.plugin' disconnected. 2017-12-30 23:55:28: netdata INFO : PLUGINSD: '/usr/libexec/netdata/plugins.d/python.d.plugin' on pid 215259 stopped after 4225 successful data collections (ENDs). 2017-12-30 23:55:28: netdata ERROR: child pid 215259 killed by signal 15. (errno 9, Bad file descriptor) 2017-12-30 23:55:28: netdata INFO : PLUGINSD: '/usr/libexec/netdata/plugins.d/python.d.plugin' thread exiting 2017-12-30 23:55:28: netdata INFO : HEALTH thread exiting 2017-12-30 23:55:29: netdata INFO : DISKSPACE thread exiting 2017-12-30 23:55:29: netdata INFO : PROC thread exiting 2017-12-30 23:55:29: netdata INFO : CGROUP thread exiting 2017-12-30 23:55:29: netdata INFO : IDLEJITTER thread exiting 2017-12-30 23:55:29: netdata INFO : netdata exiting. Bye bye... ```
k0ste commented 6 years ago

How I saw this:

selection_006

This is the same drives but in another hardware - Dell PowerEdge R730xd. I.e. I have only one Dell R530 with this issue (in same time netdata dash show metrics properly). How to debug this?

ktsaou commented 6 years ago

When source=average, netdata returns the average since the last time prometheus queried netdata. You can see this if you append &help=yes. The first line of the response will be:

# COMMENT netdata "boxe" to prometheus "10.11.13.127", source "average", last seen 110 seconds ago, time range 1514671313 to 1514671423

netdata identifies each prometheus server by its IP. If you have multiple prometheus servers querying the same netdata via the same IP (e.g. multiple prometheus in containers, or at a NATed zone), you should append &server=NAME to the URL, so that netdata will be able to keep track of each of them individually, even if they reach netdata using the same IP. Actually, it would be great if you always append this to your prometheus - just in case...

Then, since netdata averages the data between prometheus queries there should be some discrepancies, though they do not justify the diff you see.

Do the above explain your observations?

ktsaou commented 6 years ago

One way to find out if you have multiple prometheus querying netdata via the same IP is to examine /var/log/netdata/access.log. Grep the URL.

k0ste commented 6 years ago

When source=average, netdata returns the average since the last time prometheus queried netdata. You can see this if you append &help=yes.

Thanks, this was helped to find human configuration error (matched by hostname in comment string). The error is: for two instances declared same ipaddr.

Actually, it would be great if you always append this to your prometheus - just in case...

Did this.