search

GoogleCloudPlatform / prometheus-engine

Google Cloud Managed Service for Prometheus libraries and manifests.
https://g.co/cloud/managedprometheus
Apache License 2.0
189 stars 86 forks source link

amtool not working in prometheus-engine/alertmanager:v0.25.1-gmp.6-gke.0 #1066

Open suryaincbe opened 1 month ago

suryaincbe commented 1 month ago

We have enabled Google Managed Prometheus in our cluster and the control plane is in version 1.30.2-gke.1023004.

We used to view firing alerts from alertmanager pods in gmp-system namespace using the command, kubectl exec -n gmp-system alertmanager-0 -c alertmanager -- amtool --alertmanager.url=http://alertmanager:9093 alert

But now the alertmanger pods are using image prometheus-engine/alertmanager:v0.25.1-gmp.6-gke.0 in the 1.30 cluster version and we are not able use the amtool command.

alertmanager error

TheSpiritXIII commented 1 month ago

Looks like the amtool was removed in https://github.com/GoogleCloudPlatform/alertmanager/pull/81

This looks like something the team should consider adding back.

As a workaround, you could port-forward the Alertmanager container and use the embedded web UI. However, do note that some of the settings, e.g. silences, don't persist when the pod restarts!

There's also special synthetic metrics emitted from the alertmanager, which you might be able to use to see.

We apologize for the inconvenience!

suryaincbe commented 1 month ago

Looks like the amtool was removed in GoogleCloudPlatform/alertmanager#81

This looks like something the team should consider adding back.

As a workaround, you could port-forward the Alertmanager container and use the embedded web UI. However, do note that some of the settings, e.g. silences, don't persist when the pod restarts!

There's also special synthetic metrics emitted from the alertmanager, which you might be able to use to see.

We apologize for the inconvenience!

Hello,

We have many number of clusters, so to silence alerts in each of these clusters it would be difficult to use the port forward option. It would really helpful if we have the amtool.

TheSpiritXIII commented 1 month ago

Another option you can try is using the old image in conjunction with a self-deployed alertmanager. The old image is still available so you can deploy it yourself and point GMP to it.

In the meanwhile I created a PR to reintroduce amtool: https://github.com/GoogleCloudPlatform/alertmanager/pull/91