Failed to resolve alertmanager-main-1.alertmanager-operated:9094

[bruse-peng]

when i install prometheus oprator the altermager has error msg=refresh result=failure addr=alertmanager-main-1.alertmanager-operated:9094 err="1 error occurred:\n\t* Failed to resolve alertmanager-main-1.alertmanager-operated:9094: lookup alertmanager-main-1.alertmanager-operated on 10.57.252.89:53: no such host\n\n"

What did you do?

What did you expect to see?

What did you see instead? Under which circumstances?

Environment

• System information:

  insert output of uname -srm here

• Alertmanager version:

  v0.22.0

• Prometheus version:

xxxx

• Alertmanager configuration file:

  
  apiVersion: monitoring.coreos.com/v1alpha1
  kind: AlertmanagerConfig
  metadata:
  name: wechat-webhook
  namespace: monitoring
  labels:
  alertmanagerConfig: sams-qa
  spec:
  receivers:
  - name: wechat-webhook
    webhookConfigs:
      - url: http://wechat-alarm/webhook_send
        sendResolved: true
  route:
  groupWait: 30s
  groupInterval: 5m
  repeatInterval: 12h
  receiver: wechat-webhook
  routes:
    - receiver: wechat-webhook
      match:
        severity: critical

apiVersion: monitoring.coreos.com/v1 kind: Alertmanager metadata: labels: alertmanager: main name: main namespace: monitoring spec: image: quay.io/prometheus/alertmanager:v0.22.0 replicas: 3 securityContext: fsGroup: 2000 runAsNonRoot: false runAsUser: 1000 serviceAccountName: alertmanager-main version: v0.22.0 configSecret: alertmanagerConfigSelector: matchLabels: alertmanagerConfig: sams-qa

* Prometheus configuration file:

xxxxxx

* Logs:

level=info ts=2023-04-14T08:25:22.826Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml level=info ts=2023-04-14T08:25:22.826Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml level=info ts=2023-04-14T08:25:22.835Z caller=main.go:514 msg=Listening address=:9093 level=info ts=2023-04-14T08:25:22.835Z caller=tls_config.go:227 msg="TLS is disabled." http2=false level=info ts=2023-04-14T08:25:24.772Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00107379s level=info ts=2023-04-14T08:25:27.835Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml level=info ts=2023-04-14T08:25:27.836Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml level=info ts=2023-04-14T08:25:32.774Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003671261s level=warn ts=2023-04-14T08:25:37.778Z caller=cluster.go:461 component=cluster msg=refresh result=failure addr=alertmanager-main-0.alertmanager-operated:9094 err="1 error occurred:\n\t Failed to resolve alertmanager-main-0.alertmanager-operated:9094: lookup alertmanager-main-0.alertmanager-operated on 10.57.252.89:53: no such host\n\n" level=warn ts=2023-04-14T08:25:37.780Z caller=cluster.go:461 component=cluster msg=refresh result=failure addr=alertmanager-main-1.alertmanager-operated:9094 err="1 error occurred:\n\t Failed to resolve alertmanager-main-1.alertmanager-operated:9094: lookup alertmanager-main-1.alertmanager-operated on 10.57.252.89:53: no such host\n\n"

[simonpasquier]

The issue is either in the way the operator configures the statefulset or with your kubernetes DNS setup.

[MrYu1019]

Hello, I have the same problem. Have you finally solved it? I checked the coredns problem, but it was useless. There was an exception, but altermanager-main kept reporting errors, and there was really no clue.