search

prometheus / node_exporter

Exporter for machine metrics
https://prometheus.io/
Apache License 2.0
11.19k stars 2.36k forks source link

prometheus_node_exporter ntp collector failed #1427

Open liaoqiArno opened 5 years ago

liaoqiArno commented 5 years ago

OS: CentOS Linux release 7.2.1511

[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
Type=simple
User=root
Group=root
ExecStart=/app/soft/prometheus/exporters/node_exporter_current/node_exporter --collector.systemd    --collector.ntp --web.listen-address=0.0.0.0:9100 --collector.systemd.unit-whitelist=(nginx|php-fpm|prometheus-node-exporter|filebeat)\.service --log.level=info 
SyslogIdentifier=prometheus_node_exporter
Restart=always
CPUQuota=5%
MemoryAccounting=true
MemoryHigh=200M
MemoryMax=300M

[Install]
WantedBy=multi-user.target

when I see the service log ,use command: journalctl -u prometheus-node-exporter May 14 16:06:06 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-05-14T16:06:06+08:00" level=info msg="Listening on 0.0.0.0:9100" source="node_exporter.go:111" May 16 18:51:11 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-05-16T18:51:11+08:00" level=error msg="ERROR: ntp collector failed after 1.197615s: couldn't get SNTP reply: read udp 127.0.0.1:38708->127.0.0.1:123: i/o timeout" source="collector.go:132" May 17 01:15:00 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-05-17T01:15:00+08:00" level=error msg="ERROR: ntp collector failed after 1.298661s: couldn't get SNTP reply: write udp 127.0.0.1:48579->127.0.0.1:123: i/o timeout" source="collector.go:132" May 20 03:57:05 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-05-20T03:57:05+08:00" level=error msg="ERROR: ntp collector failed after 1.000280s: couldn't get SNTP reply: write udp 127.0.0.1:50363->127.0.0.1:123: i/o timeout" source="collector.go:132" May 25 03:26:35 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-05-25T03:26:35+08:00" level=error msg="ERROR: ntp collector failed after 1.100148s: couldn't get SNTP reply: read udp 127.0.0.1:36286->127.0.0.1:123: i/o timeout" source="collector.go:132" Jun 02 09:35:20 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-06-02T09:35:20+08:00" level=error msg="ERROR: ntp collector failed after 1.101965s: couldn't get SNTP reply: read udp 127.0.0.1:59231->127.0.0.1:123: i/o timeout" source="collector.go:132" Jun 03 08:20:47 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-06-03T08:20:47+08:00" level=error msg="ERROR: ntp collector failed after 1.498813s: couldn't get SNTP reply: read udp 127.0.0.1:59405->127.0.0.1:123: i/o timeout" source="collector.go:132" Jun 08 19:16:16 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-06-08T19:16:16+08:00" level=error msg="ERROR: ntp collector failed after 1.100201s: couldn't get SNTP reply: read udp 127.0.0.1:33499->127.0.0.1:123: i/o timeout" source="collector.go:132" Jun 26 06:28:33 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-06-26T06:28:33+08:00" level=error msg="ERROR: ntp collector failed after 1.299394s: couldn't get SNTP reply: read udp 127.0.0.1:50089->127.0.0.1:123: i/o timeout" source="collector.go:132" Jun 26 13:19:31 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-06-26T13:19:31+08:00" level=error msg="ERROR: ntp collector failed after 1.000196s: couldn't get SNTP reply: read udp 127.0.0.1:55898->127.0.0.1:123: i/o timeout" source="collector.go:132" Jun 30 20:18:57 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-06-30T20:18:57+08:00" level=error msg="ERROR: ntp collector failed after 1.098731s: couldn't get SNTP reply: write udp 127.0.0.1:56405->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:17:24 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:17:11+08:00" level=error msg="ERROR: ntp collector failed after 3.297278s: couldn't get SNTP reply: write udp 127.0.0.1:43096->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:19:24 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:19:24+08:00" level=error msg="ERROR: ntp collector failed after 11.099903s: couldn't get SNTP reply: read udp 127.0.0.1:38624->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:19:41 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:19:41+08:00" level=error msg="ERROR: ntp collector failed after 3.300136s: couldn't get SNTP reply: write udp 127.0.0.1:47441->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:19:47 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:19:47+08:00" level=error msg="ERROR: ntp collector failed after 9.000047s: couldn't get SNTP reply: write udp 127.0.0.1:42601->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:20:13 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:20:13+08:00" level=error msg="ERROR: ntp collector failed after 5.903447s: couldn't get SNTP reply: write udp 127.0.0.1:59662->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:21:52 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:21:41+08:00" level=error msg="ERROR: ntp collector failed after 15.301091s: couldn't get SNTP reply: read udp 127.0.0.1:60579->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:22:00 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:22:00+08:00" level=error msg="ERROR: ntp collector failed after 24.300219s: couldn't get SNTP reply: read udp 127.0.0.1:48693->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:24:58 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:24:58+08:00" level=error msg="ERROR: ntp collector failed after 14.999868s: couldn't get SNTP reply: write udp 127.0.0.1:60563->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:25:46 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:25:46+08:00" level=error msg="ERROR: ntp collector failed after 29.098297s: couldn't get SNTP reply: write udp 127.0.0.1:41081->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:26:44 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:26:44+08:00" level=error msg="ERROR: ntp collector failed after 25.761645s: couldn't get SNTP reply: write udp 127.0.0.1:43201->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:28:23 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:28:23+08:00" level=error msg="ERROR: ntp collector failed after 1.999994s: couldn't get SNTP reply: read udp 127.0.0.1:37632->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:29:08 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:29:08+08:00" level=error msg="ERROR: ntp collector failed after 16.599357s: couldn't get SNTP reply: write udp 127.0.0.1:33112->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:36:46 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:36:46+08:00" level=error msg="ERROR: ntp collector failed after 4.801277s: couldn't get SNTP reply: read udp 127.0.0.1:52083->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:41:06 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:41:06+08:00" level=error msg="ERROR: ntp collector failed after 6.100782s: couldn't get SNTP reply: write udp 127.0.0.1:55035->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:41:28 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:41:28+08:00" level=error msg="ERROR: ntp collector failed after 21.299355s: couldn't get SNTP reply: write udp 127.0.0.1:41800->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:50:25 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:50:25+08:00" level=error msg="ERROR: ntp collector failed after 6.400252s: couldn't get SNTP reply: write udp 127.0.0.1:60987->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:50:33 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:50:33+08:00" level=error msg="ERROR: ntp collector failed after 5.599039s: couldn't get SNTP reply: write udp 127.0.0.1:37936->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:50:42 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:50:42+08:00" level=error msg="ERROR: ntp collector failed after 6.899574s: couldn't get SNTP reply: write udp 127.0.0.1:38674->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:50:43 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:50:43+08:00" level=error msg="ERROR: ntp collector failed after 7.400097s: couldn't get SNTP reply: read udp 127.0.0.1:55153->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:50:46 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:50:46+08:00" level=error msg="ERROR: ntp collector failed after 11.200032s: couldn't get SNTP reply: read udp 127.0.0.1:51753->127.0.0.1:123: i/o timeout" source="collector.go:132" Jul 12 14:50:49 iZuf6izaaw3y51nmw59ou1Z prometheus_node_exporter[5853]: time="2019-07-12T14:50:49+08:00" level=error msg="ERROR: ntp collector failed after 8.299375s: couldn't get SNTP reply: write udp 127.0.0.1:34865->127.0.0.1:123: i/o timeout" source="collector.go:132

my ntpd service status: ` [root@node1 ~]# netstat -luntp | grep ntp udp 0 0 192.168.125.74:123 0.0.0.0: 13310/ntpd
udp 0 0 127.0.0.1:123 0.0.0.0: 13310/ntpd
udp 0 0 0.0.0.0:123 0.0.0.0: 13310/ntpd
udp6 0 0 :::123 ::: 13310/ntpd
[root@node1 ~]# systemctl status ntpd ● ntpd.service - Network Time Service Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2019-07-15 10:59:46 CST; 2h 59min ago Main PID: 13310 (ntpd) Memory: 692.0K CGroup: /system.slice/ntpd.service └─13310 /usr/sbin/ntpd -u ntp:ntp -g

Jul 15 10:59:46 iZuf6izaaw3y51nmw59ou1Z systemd[1]: Starting Network Time Service... Jul 15 10:59:46 iZuf6izaaw3y51nmw59ou1Z ntpd[13309]: ntpd 4.2.6p5@1.2349-o Mon Jan 25 14:27:34 UTC 2016 (1) Jul 15 10:59:46 iZuf6izaaw3y51nmw59ou1Z ntpd[13310]: proto: precision = 0.076 usec Jul 15 10:59:46 iZuf6izaaw3y51nmw59ou1Z ntpd[13310]: 0.0.0.0 c01d 0d kern kernel time sync enabled Jul 15 10:59:46 iZuf6izaaw3y51nmw59ou1Z systemd[1]: Started Network Time Service `

ntp is no problem ,what's the problem of

curantes commented 5 years ago

You need to verify that you are allowed to access the local ntpd server. This relates to #913 but that issue was with chronyd as timekeeping service. There should be some similar for ntpd.