alertmanager reliably crashes on every boot

[calestyo]

What did you do?

Every time when booting, alertmanager errors out (but works when starting later).

Environment

• System information:

Debian bookworm, Linux 6.1.0-27-amd64 x86_64

• Alertmanager version:

  alertmanager, version 0.25.0 (branch: debian/sid, revision: 0.25.0-1+b4)
  build user:       team+pkg-go@tracker.debian.org
  build date:       20230409-09:50:43
  go version:       go1.19.8
  platform:         linux/amd64

• Prometheus version:

  prometheus, version 2.42.0+ds (branch: debian/sid, revision: 2.42.0+ds-5+b5)
  build user:       team+pkg-go@tracker.debian.org
  build date:       20230518-08:49:35
  go version:       go1.19.8
  platform:         linux/amd64

• Logs:

  Nov 19 23:43:43 lcg-lrz-monitor systemd[1]: Started prometheus-alertmanager.service - Alertmanager for prometheus.
  Nov 19 23:43:43 lcg-lrz-monitor prometheus-alertmanager[616]: ts=2024-11-19T22:43:43.564Z caller=cluster.go:178 level=warn component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided"
  Nov 19 23:43:43 lcg-lrz-monitor prometheus-alertmanager[616]: ts=2024-11-19T22:43:43.569Z caller=main.go:273 level=error msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
  Nov 19 23:43:43 lcg-lrz-monitor systemd[1]: prometheus-alertmanager.service: Main process exited, code=exited, status=1/FAILURE
  Nov 19 23:43:43 lcg-lrz-monitor systemd[1]: prometheus-alertmanager.service: Failed with result 'exit-code'.
  Nov 19 23:43:43 lcg-lrz-monitor systemd[1]: prometheus-alertmanager.service: Scheduled restart job, restart counter is at 1.
  Nov 19 23:43:43 lcg-lrz-monitor systemd[1]: Stopped prometheus-alertmanager.service - Alertmanager for prometheus.
  Nov 19 23:43:43 lcg-lrz-monitor systemd[1]: Started prometheus-alertmanager.service - Alertmanager for prometheus.
  Nov 19 23:43:43 lcg-lrz-monitor prometheus-alertmanager[1141]: ts=2024-11-19T22:43:43.845Z caller=cluster.go:178 level=warn component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided"
  Nov 19 23:43:43 lcg-lrz-monitor prometheus-alertmanager[1141]: ts=2024-11-19T22:43:43.846Z caller=main.go:273 level=error msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
  Nov 19 23:43:43 lcg-lrz-monitor systemd[1]: prometheus-alertmanager.service: Main process exited, code=exited, status=1/FAILURE
  Nov 19 23:43:43 lcg-lrz-monitor systemd[1]: prometheus-alertmanager.service: Failed with result 'exit-code'.
  Nov 19 23:43:44 lcg-lrz-monitor systemd[1]: prometheus-alertmanager.service: Scheduled restart job, restart counter is at 2.
  Nov 19 23:43:44 lcg-lrz-monitor systemd[1]: Stopped prometheus-alertmanager.service - Alertmanager for prometheus.
  Nov 19 23:43:44 lcg-lrz-monitor systemd[1]: Started prometheus-alertmanager.service - Alertmanager for prometheus.
  Nov 19 23:43:44 lcg-lrz-monitor prometheus-alertmanager[1393]: ts=2024-11-19T22:43:44.249Z caller=cluster.go:178 level=warn component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided"
  Nov 19 23:43:44 lcg-lrz-monitor prometheus-alertmanager[1393]: ts=2024-11-19T22:43:44.250Z caller=main.go:273 level=error msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
  Nov 19 23:43:44 lcg-lrz-monitor systemd[1]: prometheus-alertmanager.service: Main process exited, code=exited, status=1/FAILURE
  Nov 19 23:43:44 lcg-lrz-monitor systemd[1]: prometheus-alertmanager.service: Failed with result 'exit-code'.
  Nov 19 23:43:44 lcg-lrz-monitor systemd[1]: prometheus-alertmanager.service: Scheduled restart job, restart counter is at 3.
  Nov 19 23:43:44 lcg-lrz-monitor systemd[1]: Stopped prometheus-alertmanager.service - Alertmanager for prometheus.
  Nov 19 23:43:44 lcg-lrz-monitor systemd[1]: Started prometheus-alertmanager.service - Alertmanager for prometheus.
  Nov 19 23:43:44 lcg-lrz-monitor prometheus-alertmanager[1914]: ts=2024-11-19T22:43:44.665Z caller=cluster.go:178 level=warn component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided"
  Nov 19 23:43:44 lcg-lrz-monitor prometheus-alertmanager[1914]: ts=2024-11-19T22:43:44.666Z caller=main.go:273 level=error msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
  Nov 19 23:43:44 lcg-lrz-monitor systemd[1]: prometheus-alertmanager.service: Main process exited, code=exited, status=1/FAILURE
  Nov 19 23:43:44 lcg-lrz-monitor systemd[1]: prometheus-alertmanager.service: Failed with result 'exit-code'.
  Nov 19 23:43:45 lcg-lrz-monitor systemd[1]: prometheus-alertmanager.service: Scheduled restart job, restart counter is at 4.
  Nov 19 23:43:45 lcg-lrz-monitor systemd[1]: Stopped prometheus-alertmanager.service - Alertmanager for prometheus.
  Nov 19 23:43:45 lcg-lrz-monitor systemd[1]: Started prometheus-alertmanager.service - Alertmanager for prometheus.
  Nov 19 23:43:45 lcg-lrz-monitor prometheus-alertmanager[2017]: ts=2024-11-19T22:43:45.093Z caller=cluster.go:178 level=warn component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided"
  Nov 19 23:43:45 lcg-lrz-monitor prometheus-alertmanager[2017]: ts=2024-11-19T22:43:45.094Z caller=main.go:273 level=error msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
  Nov 19 23:43:45 lcg-lrz-monitor systemd[1]: prometheus-alertmanager.service: Main process exited, code=exited, status=1/FAILURE
  Nov 19 23:43:45 lcg-lrz-monitor systemd[1]: prometheus-alertmanager.service: Failed with result 'exit-code'.
  Nov 19 23:43:45 lcg-lrz-monitor systemd[1]: prometheus-alertmanager.service: Scheduled restart job, restart counter is at 5.
  Nov 19 23:43:45 lcg-lrz-monitor systemd[1]: Stopped prometheus-alertmanager.service - Alertmanager for prometheus.
  Nov 19 23:43:45 lcg-lrz-monitor systemd[1]: prometheus-alertmanager.service: Start request repeated too quickly.
  Nov 19 23:43:45 lcg-lrz-monitor systemd[1]: prometheus-alertmanager.service: Failed with result 'exit-code'.
  Nov 19 23:43:45 lcg-lrz-monitor systemd[1]: Failed to start prometheus-alertmanager.service - Alertmanager for prometheus.
  Nov 19 23:43:45 lcg-lrz-monitor systemd[1]: prometheus-alertmanager.service: Triggering OnFailure= dependencies.
  Nov 19 23:43:45 lcg-lrz-monitor systemd[1]: Starting mail-systemd-unit-status@prometheus-alertmanager.service.service - send systemd unit status via email to `root`...
  Nov 19 23:43:47 lcg-lrz-monitor systemd[1]: mail-systemd-unit-status@prometheus-alertmanager.service.service: Deactivated successfully.
  Nov 19 23:43:47 lcg-lrz-monitor systemd[1]: Finished mail-systemd-unit-status@prometheus-alertmanager.service.service - send systemd unit status via email to `root`.
  Nov 19 23:44:11 lcg-lrz-monitor prometheus[621]: ts=2024-11-19T22:44:11.659Z caller=notifier.go:532 level=error component=notifier alertmanager=http://localhost:9093/api/v2/alerts count=2 msg="Error sending alert" err="Post \"http://localhost:9093/api/v2/alerts\": dial tcp [::1]:9093: connect: connection refused"
  Nov 19 23:44:11 lcg-lrz-monitor prometheus[621]: ts=2024-11-19T22:44:11.660Z caller=notifier.go:532 level=error component=notifier alertmanager=http://localhost:9093/api/v2/alerts count=2 msg="Error sending alert" err="Post \"http://localhost:9093/api/v2/alerts\": dial tcp [::1]:9093: connect: connection refused"
  Nov 19 23:44:53 lcg-lrz-monitor prometheus[621]: ts=2024-11-19T22:44:53.930Z caller=notifier.go:532 level=error component=notifier alertmanager=http://localhost:9093/api/v2/alerts count=9 msg="Error sending alert" err="Post \"http://localhost:9093/api/v2/alerts\": dial tcp [::1]:9093: connect: connection refused"
  Nov 19 23:45:21 lcg-lrz-monitor prometheus[621]: ts=2024-11-19T22:45:21.648Z caller=notifier.go:532 level=error component=notifier alertmanager=http://localhost:9093/api/v2/alerts count=2 msg="Error sending alert" err="Post \"http://localhost:9093/api/v2/alerts\": dial tcp [::1]:9093: connect: connection refused"
  Nov 19 23:45:21 lcg-lrz-monitor prometheus[621]: ts=2024-11-19T22:45:21.650Z caller=notifier.go:532 level=error component=notifier alertmanager=http://localhost:9093/api/v2/alerts count=2 msg="Error sending alert" err="Post \"http://localhost:9093/api/v2/alerts\": dial tcp [::1]:9093: connect: connection refused"
  Nov 19 23:46:31 lcg-lrz-monitor prometheus[621]: ts=2024-11-19T22:46:31.648Z caller=notifier.go:532 level=error component=notifier alertmanager=http://localhost:9093/api/v2/alerts count=2 msg="Error sending alert" err="Post \"http://localhost:9093/api/v2/alerts\": dial tcp [::1]:9093: connect: connection refused"
  Nov 19 23:46:31 lcg-lrz-monitor prometheus[621]: ts=2024-11-19T22:46:31.650Z caller=notifier.go:532 level=error component=notifier alertmanager=http://localhost:9093/api/v2/alerts count=2 msg="Error sending alert" err="Post \"http://localhost:9093/api/v2/alerts\": dial tcp [::1]:9093: connect: connection refused"
  Nov 19 23:46:53 lcg-lrz-monitor prometheus[621]: ts=2024-11-19T22:46:53.930Z caller=notifier.go:532 level=error component=notifier alertmanager=http://localhost:9093/api/v2/alerts count=9 msg="Error sending alert" err="Post \"http://localhost:9093/api/v2/alerts\": dial tcp [::1]:9093: connect: connection refused"

So it seems there are a number of errors involved here:

• couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided
• create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided

Not really sure what it means by "private IP" (or why it should need any), any normal UNIX daemon typically binds to the wildcard address if no specific bind addresses are given.
Also, the service is pulled in by multi-user.target and at that time any networking (including the statically configured global IPs) are long up.

• Error sending alert" err="Post \"http://localhost:9093/api/v2/alerts\": dial tcp [::1]:9093: connect: connection refused These are also a bit strange, IMO,... at least is they'd cause the daemon to exit. I mean it should rather be clear that prometheus may not yet be running.

Anyway, if I start the daemon a bit later, it works just fine.

Cheers, Chris,

[grobinson-grafana]

Hi!

Alertmanager is crashing because it cannot get the information it needs to initialize the cluster for high availability mode. The error means it cannot find a private IP address for the system, which it advertises to other Alertmanagers in the same cluster.

If your system does not have a private IP address, and/or you do not need high availability mode, you can disable it with the following argument:

--cluster.listen-address=""

Error sending alert" err="Post \"http://localhost:9093/api/v2/alerts\": dial tcp [::1]:9093: connect: connection refused

Your Prometheus can't send alerts to Alertmanager because it's crash looping.