Closed sed-i closed 3 months ago
While testing https://github.com/canonical/cos-proxy-operator/pull/136, prometheus pod got rescheduled, and all alerts were gone.
juju show-unit has alerts:
juju show-unit
prom/0: workload-version: 2.50.1 opened-ports: [] charm: ch:amd64/focal/prometheus-k8s-189 leader: true life: alive relation-info: - relation-id: 6 endpoint: metrics-endpoint cross-model: true related-endpoint: downstream-prometheus-scrape application-data: alert_rules: '{"groups": [{"name": "juju_luca_30d8d8d_ubuntu_0_alert_rules", "rules": [{"alert": "CheckConntrackNrpeAlert", "expr": "avg_over_time(command_status{juju_unit=''ubuntu/0'',command=''check_conntrack''}[15m]) > 1 or (absent_over_time(command_status{juju_unit=''ubuntu/0'',command=''check_conntrack''}[10m]) # ...
and the alerts are indeed on disk:
❯ jsshc prometheus prom/0 cat /etc/prometheus/rules/juju_juju_luca_30d8d8d_ubuntu_0_alert_rules_metrics-endpoint_6.rules groups: - name: juju_luca_30d8d8d_ubuntu_0_alert_rules rules: - alert: CheckConntrackNrpeAlert annotations: description: "Check provided by nrpe_exporter in model {{ $labels.juju_model\ \ }} is failing.\nFailing check = {{ $labels.command }}\nUnit = {{ $labels.juju_unit\ # ...
but the webui shows nothing:
❯ curl 10.1.166.113:9090/api/v1/alerts {"status":"success","data":{"alerts":[]}}
kubectl logs seem fine:
2024-05-28T13:30:04.558Z [prometheus] ts=2024-05-28T13:30:04.558Z caller=main.go:1324 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-05-28T13:30:04.559Z [prometheus] ts=2024-05-28T13:30:04.559Z caller=main.go:1361 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=1.795905ms db_storage=780ns remote_storage=670ns web_handler=160ns query_engine=520ns scrape=155.801µs scrape_sd=38.728µs notify=1.13µs notify_sd=510ns rules=1.142212ms tracing=5.54µs 2024-05-28T13:30:04.559Z [prometheus] ts=2024-05-28T13:30:04.559Z caller=main.go:1103 level=info msg="Server is ready to receive web requests."
Deploy this bundle in a k8s model:
bundle: kubernetes saas: remote-2c369e446bcc43d58ca9ab95eb920c5c: {} applications: prom: charm: prometheus-k8s channel: latest/edge revision: 189 resources: prometheus-image: 144 scale: 1 trust: true --- # overlay.yaml applications: prom: offers: prom: endpoints: - metrics-endpoint - receive-remote-write acl: admin: admin
And this bundle in a lxd model:
series: jammy saas: prom: url: j34:admin/pg.prom applications: cos-proxy: charm: cos-proxy channel: latest/edge num_units: 1 nrpe: charm: nrpe channel: latest/edge revision: 122 ubuntu: charm: ubuntu channel: stable revision: 24 num_units: 1 relations: - - ubuntu:juju-info - nrpe:general-info - - cos-proxy:monitors - nrpe:monitors - - cos-proxy:downstream-prometheus-scrape - prom:metrics-endpoint
Then refresh cos-proxy.
Controller Model User Access Cloud/Region Models Nodes HA Version j34 pg admin superuser microk8s/localhost 2 1 - 3.4.2 lxd2* luca admin superuser localhost/localhost 2 1 none 3.1.7
❯ jssl prom/0 --days=11 Time Type Status Message 28 May 2024 09:12:52-04:00 juju-unit allocating 28 May 2024 09:12:52-04:00 workload waiting installing agent 28 May 2024 09:13:17-04:00 workload waiting agent initialising 28 May 2024 09:13:50-04:00 workload maintenance installing charm software 28 May 2024 09:13:50-04:00 juju-unit executing running install hook 28 May 2024 09:13:54-04:00 juju-unit executing running prometheus-peers-relation-created hook 28 May 2024 09:13:56-04:00 juju-unit executing running leader-elected hook 28 May 2024 09:13:59-04:00 workload active 28 May 2024 09:14:00-04:00 juju-unit executing running prometheus-pebble-ready hook 28 May 2024 09:14:03-04:00 juju-unit executing running database-storage-attached hook 28 May 2024 09:14:05-04:00 juju-unit executing running config-changed hook 28 May 2024 09:14:09-04:00 workload waiting Waiting for resource limit patch to apply 28 May 2024 09:14:59-04:00 juju-unit error hook failed: "start" 28 May 2024 09:15:04-04:00 juju-unit executing running start hook 28 May 2024 09:15:08-04:00 juju-unit executing running prometheus-pebble-ready hook 28 May 2024 09:15:21-04:00 juju-unit idle 28 May 2024 09:15:51-04:00 juju-unit executing running metrics-endpoint-relation-created hook 28 May 2024 09:15:54-04:00 juju-unit executing running metrics-endpoint-relation-changed hook 28 May 2024 09:15:56-04:00 juju-unit executing running metrics-endpoint-relation-joined hook for remote-2c369e446bcc43d58ca9ab95eb920c5c/0 28 May 2024 09:15:58-04:00 juju-unit executing running metrics-endpoint-relation-changed hook for remote-2c369e446bcc43d58ca9ab95eb920c5c/0 28 May 2024 09:16:01-04:00 juju-unit executing running metrics-endpoint-relation-changed hook 28 May 2024 09:16:03-04:00 juju-unit idle 28 May 2024 09:19:22-04:00 workload active 28 May 2024 09:22:23-04:00 workload maintenance stopping charm software 28 May 2024 09:22:23-04:00 juju-unit executing running stop hook 28 May 2024 09:22:24-04:00 workload active 28 May 2024 09:22:27-04:00 juju-unit idle 28 May 2024 09:22:27-04:00 workload maintenance 28 May 2024 09:30:01-04:00 juju-unit executing running upgrade-charm hook 28 May 2024 09:30:06-04:00 juju-unit executing running config-changed hook 28 May 2024 09:30:09-04:00 juju-unit executing running start hook 28 May 2024 09:30:11-04:00 juju-unit executing running prometheus-pebble-ready hook 28 May 2024 09:30:14-04:00 juju-unit executing running metrics-endpoint-relation-changed hook 28 May 2024 09:30:29-04:00 juju-unit idle 28 May 2024 23:32:26-04:00 workload active
No response
Oops, was looking at api/v1/alerts instead of api/v1/rules. Closing!
api/v1/alerts
api/v1/rules
Bug Description
While testing https://github.com/canonical/cos-proxy-operator/pull/136, prometheus pod got rescheduled, and all alerts were gone.
juju show-unit
has alerts:and the alerts are indeed on disk:
but the webui shows nothing:
kubectl logs seem fine:
To Reproduce
Deploy this bundle in a k8s model:
And this bundle in a lxd model:
Then refresh cos-proxy.
Environment
Relevant log output
Additional context
No response