search

GoogleCloudPlatform / prometheus-engine

Google Cloud Managed Service for Prometheus libraries and manifests.
https://g.co/cloud/managedprometheus
Apache License 2.0
191 stars 89 forks source link

[DRAFT] Prevent pod restarts on startup #920

Open pintohutch opened 5 months ago

pintohutch commented 5 months ago

It is tempting to only rely on defaulting webhooks to ensure any changes to the OperatorConfig updates the .collection.externalLabels and rules.externalLabels fields with the default project, location, and cluster labels.

However, for many cases, where an OperatorConfig already exists and the operator is restarted with this change, a webhook will not be invoked and the change to the generated Prometheus config's external_labels will never happen.

So we add the defaulting webhook as well as implicit logic to fill in the generated config external_labels at runtime.

Another approach could be to omit the mutating webhook entirely, however if we can reflect the state of the config in our custom resources, we should. This gives users a more clear and explicit sense of what is being configured.

bwplotka commented 4 months ago

I like this approach, but we have to manually (or/and automatically) tested for two cases - new and existing OperatorConfig.

(Not on it, don't have time now, help wanted)