search

canonical / cos-tool

Apache License 2.0
3 stars 2 forks source link

cos-tool fails to validate tempo rules #13

Open sed-i opened 9 months ago

sed-i commented 9 months ago

In https://github.com/canonical/tempo-k8s-operator/pull/11 we introduced alert rules taken from https://github.com/grafana/tempo/tree/main/operations/tempo-mixin-compiled but cos-tool fails to validate them:

$ ./cos-tool-amd64 --format logql validate loki_alert_rules/alerts.yaml 
error validating loki_alert_rules/alerts.yaml: [parse error at line 1, col 1: syntax error: unexpected IDENTIFIER
could not parse expression for record '' in group 'tempo_alerts'
$ ./cos-tool-amd64 --format logql validate loki_alert_rules/rules.yaml 
error validating loki_alert_rules/rules.yaml: [parse error at line 1, col 1: syntax error: unexpected IDENTIFIER
could not parse expression for record 'cluster_namespace_job_route:tempo_request_duration_seconds:99quantile' in group 'tempo_rules'
sed-i commented 9 months ago

Related a few charms (tempo, traefik) all of which have built-in loki rules, and only traefik's rules reached grafana agent's relation data.

bundle: kubernetes
applications:
  ga:
    charm: grafana-agent-k8s
    channel: edge
    scale: 1
  tempo:
    charm: tempo-k8s
    channel: edge
    scale: 1
  trfk:
    charm: traefik-k8s
    channel: edge
    series: focal
    scale: 1
relations:
- - tempo:logging
  - ga:logging-provider
- - trfk:logging
  - ga:logging-provider
ga/0:
  workload-version: 0.35.2
  opened-ports: []
  charm: ch:amd64/jammy/grafana-agent-k8s-60
  leader: true
  life: alive
  relation-info:
  - relation-id: 20
    endpoint: logging-consumer
    related-endpoint: logging
    application-data:
      promtail_binary_zip_url: '{"amd64": {"filename": "promtail-static-amd64", "zipsha":
        "543e333b0184e14015a42c3c9e9e66d2464aaa66eca48b29e185a6a18f67ab6d", "binsha":
        "17e2e271e65f793a9fbe81eab887b941e9d680abe82d5a0602888c50f5e0cac9", "url":
        "https://github.com/canonical/loki-k8s-operator/releases/download/promtail-v2.5.0/promtail-static-amd64.gz"}}'
    related-units:
      loki/0:
        in-scope: true
        data:
          egress-subnets: 10.152.183.32/32
          endpoint: '{"url": "http://loki-0.loki-endpoints.welcome-k8s.svc.cluster.local:3100/loki/api/v1/push"}'
          ingress-address: 10.152.183.32
          private-address: 10.152.183.32
          public_address: loki-0.loki-endpoints.welcome-k8s.svc.cluster.local
  - relation-id: 19
    endpoint: logging-provider
    related-endpoint: logging
    application-data:
      alert_rules: '{}'
      metadata: '{"model": "welcome-k8s", "model_uuid": "bf35e241-d1b5-4613-810a-759a7065d3ca",
        "application": "tempo", "unit": "tempo/0", "charm_name": "tempo-k8s"}'
    related-units:
      tempo/0:
        in-scope: true
        data:
          egress-subnets: 10.152.183.141/32
          ingress-address: 10.152.183.141
          private-address: 10.152.183.141
  - relation-id: 24
    endpoint: logging-provider
    related-endpoint: logging
    application-data:
      alert_rules: '{"groups": [{"name": "welcome-k8s_bf35e241_trfk_error_level_logs_found_alerts",
        "rules": [{"alert": "ErrorLevelLogsFound", "annotations": {"summary": "error
        level logs found."}, "expr": "rate({job=~\".+\"} |= \"level=error\"[1m]) >
        0", "for": "1m", "labels": {"juju_application": "trfk", "juju_charm": "traefik-k8s",
        "juju_model": "welcome-k8s", "juju_model_uuid": "bf35e241-d1b5-4613-810a-759a7065d3ca",
        "severity": "critical"}}]}]}'
      metadata: '{"model": "welcome-k8s", "model_uuid": "bf35e241-d1b5-4613-810a-759a7065d3ca",
        "application": "trfk", "unit": "trfk/0", "charm_name": "traefik-k8s"}'
    related-units:
      trfk/0:
        in-scope: true
        data:
          egress-subnets: 10.43.8.188/32
          ingress-address: 10.43.8.188
          private-address: 10.43.8.188