canonical / grafana-agent-k8s-operator

https://charmhub.io/grafana-agent-k8s
Apache License 2.0
8 stars 18 forks source link

Add alert rules for conntrack #188

Closed rgildein closed 1 year ago

rgildein commented 1 year ago

Context

Moving network NRPE checks from charm-nrpe.

Testing Instructions

Tested with

rule_files:
  - conntrack.rules

evaluation_interval: 1m

tests:
  # disk usage prediction
  - interval: 1m
    input_series:
      - series: 'node_nf_conntrack_entries{instance="test-model_1234_test-app_test-app/0"}'
        values: '50x25 850x10 50x25'
      - series: 'node_nf_conntrack_entries_limit{instance="test-model_1234_test-app_test-app/0"}'
        values: '1000x60'
      - series: 'node_nf_conntrack_entries{instance="test-model_1234_test-app_test-app/1"}'
        values: '50x25 150x10 50x25'
      - series: 'node_nf_conntrack_entries_limit{instance="test-model_1234_test-app_test-app/1"}'
        values: '1000x60'
    promql_expr_test:
      - expr: node_nf_conntrack_entries / node_nf_conntrack_entries_limit
        eval_time: 30m
        exp_samples:
          - labels: '{instance="test-model_1234_test-app_test-app/0"}'
            value: 0.85
          - labels: '{instance="test-model_1234_test-app_test-app/1"}'
            value: 0.15
    alert_rule_test:
      - eval_time: 10m
        alertname: HostConntrackLimit
        exp_alerts: []  # no alert
      - eval_time: 28m
        alertname: HostConntrackLimit
        exp_alerts:
          - exp_labels:
              severity: warning
              instance: test-model_1234_test-app_test-app/0
            exp_annotations:
              summary: Host conntrack reached 85% (instance test-model_1234_test-app_test-app/0)
              description: >-
                Host conntrack reached 85% usage.
                  VALUE = 85
                  LABELS = map[instance:test-model_1234_test-app_test-app/0]

and promtool

x1:➜  prometheus_alert_rules git:(nrpe/contrack-aler-rules) ✗ promtool test rules ./test_conntrack.yaml
Unit Testing:  ./test_conntrack.yaml
  SUCCESS
                                                                                                  [0.17s]

Release Notes