Discrepancy between alerts firing in prometheus and those reported in alertmanager

[nishant-dash]

Bug Description

In a fresh deployment of an environment, we have a COS with Charmed K8s and Charmed Kubeflow.

Currently we see 13 alerts firing in prometheus, however only 4 show up in alertmanager.

To Reproduce

N/A

Environment

This is with Charmed K8s 1.29/beta and Charmed Kubeflow 1.8/stable on top of Juju 3.1.6

App                       Version  Status  Scale  Charm                         Channel        Rev  Address         Exposed  Message
alertmanager              0.25.0   active      1  alertmanager-k8s              stable          96  REMOVED   no       
catalogue                          active      1  catalogue-k8s                 stable          31  REMOVED   no       
grafana                   9.2.1    active      1  grafana-k8s                   stable          93  REMOVED  no       
loki                      2.7.4    active      1  loki-k8s                      stable         105  REMOVED  no       
prometheus                2.47.2   active      1  prometheus-k8s                stable         156  REMOVED   no       
prometheus-scrape-config  n/a      active      1  prometheus-scrape-config-k8s  latest/stable   44  REMOVED  no       
traefik                   2.10.4   active      1  traefik-k8s                   stable         166  REMOVED  no       

Relevant log output

N/A

Additional context

No response

[lucabello]

This might be due to https://github.com/canonical/alertmanager-k8s-operator/issues/207; if not, we should investigate further :)

[sed-i]

Hi @nishant-dash, Now with #212 merged, would you be able to re-test with revision 98 from edge?

[nishant-dash]

it seems like upgrading to rev 98 did the trick, almost certain because of https://github.com/canonical/alertmanager-k8s-operator/issues/207 like @lucabello mentioned

[lucabello]

Awesome, closing the issue then :)