grafana / k8s-monitoring-helm

Apache License 2.0
139 stars 58 forks source link

Improve `helm test` so it can assert values #257

Closed petewall closed 5 months ago

petewall commented 7 months ago

Currently, Helm test only checks that metrics or logs exist. I'd love to add an expected result to the query to assert things like the number of nodes is accurate.

Also, would love to do something like:

  1. Enable scrape_samples_scraped in Grafana Agent allow list.
  2. Run the query count_over_time(scrape_samples_scraped[1m]) to calculate DPM
  3. Check that the value should be 1
petewall commented 5 months ago

New queries schema:

- query: <string>
  type: <enum:["promql"(def), "logql", "traceql"]>
  expect:
    operator: <enum:["<","<=","="(def),">=",">","!="]>
    value: <number>

example:

- query: count(kube_node_info{cluster="my-3-node-cluster"})
  type: "promql"
  expect:
    value: 3
- query: up{job="integrations/node_exorter}
  type: "promql"
  expect:
    operator: !=
    value: 0
petewall commented 5 months ago

I just realized that this might work for metrics, but it'd likely be different for logs and traces... TBD on those.

skl commented 5 months ago

lgtm @petewall 👍