search

canonical / cos-proxy-operator

https://charmhub.io/cos-proxy
Apache License 2.0
2 stars 12 forks source link

nrpe checks fail after adding relabel_configs in prometheus #157

Open ahmadfsbd1 opened 2 months ago

ahmadfsbd1 commented 2 months ago

Bug Description

After adding relabel_configs in cos-prometheus through prometheus-scrape-config-k8s charm, the nrpe checks fail. When the relabel_config is removed, the nrpe checks and alerts will work as expected.

To Reproduce

  1. Deploy the standard cos
  2. Deploy nrpe charm in a separate model with relation to cos-proxy forwarding to cos-prometheus
  3. Add the relabel_config to the cos-prometheus using the prometheus-scrape-config-k8s-charm. relabel_configs: |-
    • target_label: 'alerting_system' replacement: 'nfvi-cos'
    • target_label: 'site_identifier' replacement: 'site1'
  4. Observe that after the prometheus has resstarted and settled, the nrpe checks are failing.

Environment

juju 3.4/stable, MAAS 3.4/stable

alertmanager alertmanager-k8s latest/edge 128
catalogue catalogue-k8s latest/stable 43
grafana grafana-k8s latest/stable loki loki-k8s latest/stable prometheus prometheus-k8s latest/stable scrape-interval-config-monitors prometheus-scrape-config-k8s latest/edge traefik traefik-k8s latest/stable 194

cos-proxy-monitors cos-proxy latest/edge 101 nrpe nrpe latest/stable

Relevant log output

See the images in 'To Reproduce'

Additional context

No response

simskij commented 2 months ago

cc @sed-i

sed-i commented 1 month ago

@simskij was this the issue you fixed by using metrics_relabel_config instead of relable_configs?