canonical / prometheus-k8s-operator

https://charmhub.io/prometheus-k8s
Apache License 2.0
21 stars 35 forks source link

Validate scrape configs/targets from incoming relations #314

Closed rbarry82 closed 1 year ago

rbarry82 commented 2 years ago

The addition of a config validation action provides a good way to validate a complete Prometheus configuration, but it is only useful as a all-or-nothing check, where a potentially invalid configuration must either not be pushed to the workload, which may also miss other changes in relations, or to apply it and block Prometheus entirely, forcing the use of the action.

In addition, on an upgrade, pod restart, or other scenario whether the workload container is not running, it is not possible to pre-preemptively validate the configuration, since promtool will not be accessible.

Similar to cos-tool validating alerts, we could extract the portions of promtool used to evaluate rules/targets and validate them in an isolated fashion, sending events back as appropriate (again, similar to the alert rule validation).

dstathis commented 1 year ago

I see no reason to extract code from promtool as it already ships with the prometheus rock. We can use the full tool directly.