search

nodejs / build

Better build and test infra for Node.
505 stars 165 forks source link

Influxdb address needs updating #3228

Closed richardlau closed 1 year ago

richardlau commented 1 year ago

Just spotted this on www:

# tail /var/log/syslog
Mar 16 15:14:00 infra-digitalocean-ubuntu1604-x64-1 telegraf[20610]: 2023-03-16T15:14:00Z W! [outputs.influxdb] Metric buffer overflow; 77 metrics have been dropped
Mar 16 15:14:00 infra-digitalocean-ubuntu1604-x64-1 kernel: [48323787.442327] [UFW BLOCK] IN=eth0 OUT= MAC=ba:80:a7:7e:12:f6:fe:00:00:00:01:01:08:00 SRC=222.246.64.221 DST=142.93.60.201 LEN=40 TOS=0x00 PREC=0x00 TTL=47 ID=2482 PROTO=TCP SPT=38983 DPT=23 WINDOW=48231 RES=0x00 SYN URGP=0
Mar 16 15:14:05 infra-digitalocean-ubuntu1604-x64-1 telegraf[20610]: 2023-03-16T15:14:05Z E! [outputs.influxdb] When writing to [http://147.75.88.62:8086]: failed doing req: Post "http://147.75.88.62:8086/write?db=monitoring": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Mar 16 15:14:05 infra-digitalocean-ubuntu1604-x64-1 telegraf[20610]: 2023-03-16T15:14:05Z E! [agent] Error writing to outputs.influxdb: could not write any address
Mar 16 15:14:05 infra-digitalocean-ubuntu1604-x64-1 systemd[1]: Started Session 600902 of user root.
Mar 16 15:14:05 infra-digitalocean-ubuntu1604-x64-1 systemd[1]: Started Session 600903 of user root.
Mar 16 15:14:10 infra-digitalocean-ubuntu1604-x64-1 telegraf[20610]: 2023-03-16T15:14:10Z W! [outputs.influxdb] Metric buffer overflow; 77 metrics have been dropped
Mar 16 15:14:13 infra-digitalocean-ubuntu1604-x64-1 kernel: [48323800.483787] [UFW BLOCK] IN=eth0 OUT= MAC=ba:80:a7:7e:12:f6:fe:00:00:00:01:01:86:dd SRC=2a05:d01c:0b43:8a10:34ef:44de:f55b:e9eb DST=2604:a880:0400:00d1:0000:0000:0a3c:f001 LEN=64 TC=0 HOPLIMIT=237 FLOWLBL=0 PROTO=TCP SPT=21345 DPT=5038 WINDOW=1024 RES=0x00 SYN URGP=0
Mar 16 15:14:15 infra-digitalocean-ubuntu1604-x64-1 telegraf[20610]: 2023-03-16T15:14:15Z E! [outputs.influxdb] When writing to [http://147.75.88.62:8086]: failed doing req: Post "http://147.75.88.62:8086/write?db=monitoring": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Mar 16 15:14:15 infra-digitalocean-ubuntu1604-x64-1 telegraf[20610]: 2023-03-16T15:14:15Z E! [agent] Error writing to outputs.influxdb: could not write any address

147.75.88.62 is the old IP address of the grafana server, moved as part of https://github.com/nodejs/build/issues/3108.

richardlau commented 1 year ago

On www I've updated the IP address in /etc/telegraf/telegraf.conf and restarted the service (systemctl restart telegraf.service). Will need to do the same for the other servers connected to grafana.

richardlau commented 1 year ago

Updated unencrypted, ci, ci-release and gh-bot. As an aside, this has triggered a low disk space alert for www in #nodejs-build-infra-alerts on the OpenJS Foundation Slack.