Alerting: External Alertmanager stuck in pending state

[gillesdemey]

After configuring my Grafana Cloud instance as an external alertmanager it seems to be stuck in pending state. After some initial digging it seems like the activeAlertmanager from /api/v1/ngalert/alertmanagers responds with a sightly different URL.

{
  "status": "success",
  "data": {
    "activeAlertManagers": [
      {
        "url": "https://alertmanager-eu-west-0.grafana.net/alertmanager/api/v2/alerts"
      }
    ],
    "droppedAlertManagers": []
  }
}

And the URL that is configured is https://alertmanager-eu-west-0.grafana.net/ but gets turned in to https://alertmanager-eu-west-0.grafana.net/alertmanager/api/v2/alerts here

https://github.com/grafana/grafana/blob/ef09119ed36154dc70da02d00a1975f898ebb848/public/app/features/alerting/unified/hooks/useExternalAmSelector.ts#L45

This path mismatch will result in the AM stuck in pending state since it'll never be in either active or dropped alertmanagers.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last year. It will be closed in 30 days if no further activity occurs. Please feel free to leave a comment if you believe the issue is still relevant. Thank you for your contributions!

[michaelruigrok]

There still seems to be difficulty in getting a provisional configuration to use an external alertmanager, without having to set contact points and notifiers manually.

for instance, https://github.com/grafana/helm-charts/issues/3018 has several thumbs-ups, I believe this is an extended issue.

I expect the intended workflow is to use an alertmanager datasource via the internal grafana alertmanager, rather than setting a global alertmanagersChoice (as per https://github.com/grafana/grafana/issues/80349). However with this functionality broken, users search for another way to configure this.

[gillesdemey]

That seem orthogonal, I understand why users would want alertmanagersChoice to be provisioned and the request to disabled / hide the internal Alertmanager but that's unrelated to external AMs getting stuck in pending.

[michaelruigrok]

That seem orthogonal, I understand why users would want alertmanagersChoice to be provisioned and the request to disabled / hide the internal Alertmanager but that's unrelated to external AMs getting stuck in pending.

I just meant they're related as different mechanisms to send alerts to external alertmanagers. Currently neither works, so there's no good provisional method of using an external alertmanager.

I wanted to highlight this issue's importance and remove the stale tag to hopefully increase this issue's priority.