The addition of a config validation action provides a good way to validate a complete Prometheus configuration, but it is only useful as a all-or-nothing check, where a potentially invalid configuration must either not be pushed to the workload, which may also miss other changes in relations, or to apply it and block Prometheus entirely, forcing the use of the action.
In addition, on an upgrade, pod restart, or other scenario whether the workload container is not running, it is not possible to pre-preemptively validate the configuration, since promtool will not be accessible.
Similar to cos-tool validating alerts, we could extract the portions of promtool used to evaluate rules/targets and validate them in an isolated fashion, sending events back as appropriate (again, similar to the alert rule validation).
The addition of a config validation action provides a good way to validate a complete Prometheus configuration, but it is only useful as a all-or-nothing check, where a potentially invalid configuration must either not be pushed to the workload, which may also miss other changes in relations, or to apply it and block Prometheus entirely, forcing the use of the action.
In addition, on an upgrade, pod restart, or other scenario whether the workload container is not running, it is not possible to pre-preemptively validate the configuration, since
promtool
will not be accessible.Similar to
cos-tool
validating alerts, we could extract the portions ofpromtool
used to evaluate rules/targets and validate them in an isolated fashion, sending events back as appropriate (again, similar to the alert rule validation).