search

canonical / cos-configuration-k8s-operator

This charmed operator for Kubernetes enables you to provide configurations to various components of the Canonical Observability Stack (COS) bundle.
https://charmhub.io/cos-configuration-k8s
Apache License 2.0
4 stars 6 forks source link

Juju topology labels shouldn't be added to loki rule files #44

Closed sed-i closed 1 year ago

sed-i commented 1 year ago

Bug Description

The charm currently uses LokiPushApiConsumer to forward rule files to loki, but that class currently auto adds topology labels to the alerts.

https://github.com/canonical/cos-configuration-k8s-operator/blob/a0902afca80ee2ea6b54abd086d52198df39c5f5/src/charm.py#L137-L142

To Reproduce

juju deploy --channel=edge cos-configuration-k8s cos-config --config git_repo=https://github.com/canonical/cos-configuration-k8s-operator.git --config git_branch=main --config loki_alert_rules_path=tests/samples/loki_alert_rules
juju deploy --channel=edge loki-k8s loki
juju relate loki cos-config

juju ssh --container loki loki/0 grep -r "juju_" /loki/rules/fake

Environment

Model    Controller           Cloud/Region        Version  SLA          Timestamp
welcome  charm-dev-batteries  microk8s/localhost  3.0.2    unsupported  12:55:10-05:00

App         Version  Status  Scale  Charm                  Channel  Rev  Address         Exposed  Message
cos-config  3.5.0    active      1  cos-configuration-k8s  edge      18  10.152.183.195  no       
loki        2.4.1    active      1  loki-k8s               edge      65  10.152.183.124  no

Relevant log output

NTA

Additional context

No response

rbarry82 commented 1 year ago

LokiPushApiConsumer should add topology labels, so it behaves the same as PrometheusRemoteWrite, or we'll cause problems with the subordinate Grafana Agent (which will never be able to have principal charms contribute rules which have the correct labels, which are further used by Loki to modify the expressions themselves).

Instead, cos-config needs a workaround/arg for this, similar to the way it has one for Grafana Dashboards.