Alertmanager HA external URL

[nelsonfassis]

What did you do? This is my Alertmanager Statefulset so far.

spec:
      containers:
      - name: alertmanager
        image: prom/alertmanager:v0.12.0
        args:
          - '--config.file=/etc/alertmanager/config.yml'
          - '--storage.path=/alertmanager'
          - '--web.external-url=http://myk8sdomain.com:31080/alertmanager'
          - '--mesh.listen-address=:6783'
          - '--mesh.peer=alertmanager-mesh'

Alertmanager service:

apiVersion: v1
kind: Service
metadata:
  name: alertmanager
  labels:
    app: alertmanager
spec:
  ports:
  - port: 9093
    name: alertmanager
  clusterIP: None
  selector:
    app: alertmanager

Prometheus configuration:

      alerting:
        alertmanagers:
        - scheme: http
          static_configs:
          - targets:
             - alertmanager-0.alertmanager:9093
             - alertmanager-1.alertmanager:9093

What did you expect to see? I expected Prometheus (as many instances as needed) to fire alerts to Alertmanager that would dedup those alerts and fire alerts to my slack. What did you see instead? Under which circumstances? If I have --web.external-url set:

level=error ts=2018-02-26T20:22:13.937418135Z caller=notifier.go:444 component=notifier alertmanager=http://alertmanager-0.alertmanager:9093/api/v1/alerts count=1 msg="Error sending alert" err="bad response status 404 Not Found"

If I comment out --web.external-url: Alerts are properly sent to Alertmanager which fires to my Slack. But when I click on the link in the slack alert, it opens ( http://alertmanager-1:9093/#/alerts?receiver=slack_alert ) instead of (http://myk8sdomain.com:31080/alertmanager/alerts?receiver=slack_alert').

I've also tried using nodePort as service to make sure Ingress wasn't the problem here, same result.

Environment

• System information:

  insert output of uname -srm here

• Prometheus version: Version | 2.0.0

• Alertmanager version:

0.12.0

• Logs:

  level=info ts=2018-02-26T20:21:48.984925901Z caller=main.go:394 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
  level=info ts=2018-02-26T20:21:48.989696818Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
  level=info ts=2018-02-26T20:21:48.991574227Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
  level=info ts=2018-02-26T20:21:48.993095013Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
  level=info ts=2018-02-26T20:21:48.994535886Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
  level=info ts=2018-02-26T20:21:48.995950911Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
  level=info ts=2018-02-26T20:21:48.997557533Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
  level=info ts=2018-02-26T20:21:48.998983519Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
  level=info ts=2018-02-26T20:21:49.000719298Z caller=kubernetes.go:100 component="target manager" discovery=k8s msg="Using pod service account via in-cluster config"
  level=info ts=2018-02-26T20:21:49.00367165Z caller=main.go:371 msg="Server is ready to receive requests."
  level=error ts=2018-02-26T20:22:13.937418135Z caller=notifier.go:444 component=notifier alertmanager=http://alertmanager-0.alertmanager:9093/api/v1/alerts count=1 msg="Error sending alert" err="bad response status 404 Not Found"
  level=error ts=2018-02-26T20:22:13.938614548Z caller=notifier.go:444 component=notifier alertmanager=http://alertmanager-1.alertmanager:9093/api/v1/alerts count=1 msg="Error sending alert" err="bad response status 404 Not Found" 

[nelsonfassis]

Execing into my pod and doing a wget http://alertmanager-1.alertmanager:9093/alertmanager/api/v1/alerts works, using path_prefix on my prometheus configmap didn't help at all: `

alerting:
        alertmanagers:
        - scheme: http
          path_prefix: "/alertmanager"
          static_configs:
          - targets:
              - alertmanager-0.alertmanager:9093
              - alertmanager-1.alertmanager:9093

`

[simonpasquier]

Have you tried adding --web.route-prefix="/" to AlertManager flags? When setting --web.external-url=http://myk8sdomain.com:31080/alertmanager, AlertManager will default web.route-prefix to the external URL's path (eg /alertmanager).

[pathania29]

Any Answer for above problem. I am getting same error

[abhigupta1207]

@pathania29 Can you help me in configuring alertmanager HA on Kubernetes?