search

GoogleCloudPlatform / prometheus-engine

Google Cloud Managed Service for Prometheus libraries and manifests.
https://g.co/cloud/managedprometheus
Apache License 2.0
191 stars 89 forks source link

Can't create PodMonitoring on Autopilot 1.25 #782

Closed rafaelkelly closed 7 months ago

rafaelkelly commented 7 months ago

Version is 1.25.15-gke.1115000. Get the following error: Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "default.podmonitorings.gmp-operator.gmp-system.monitoring.googleapis.com": failed to call webhook: Post "https://gmp-operator.gmp-system.svc:443/default/monitoring.googleapis.com/v1/podmonitorings?timeout=10s": service "gmp-operator" not found I see the namespace is gke-gmp-system instead of gmp-system. I don't see a way to tell PodMonitoring what the operator namespace is. What the recommendation in this case?

rafaelkelly commented 7 months ago

I think it's working now. Turns out there were 2 webhooks gmp-operator.gke-gmp-system.monitoring.googleapis.com and gmp-operator.gmp-system.monitoring.googleapis.com (maybe leftover from an upgrade?). Deleting the gmp-system ones allowed me to create the PodMonitoring I was having trouble with :)

maxamins commented 7 months ago

Hey Rafael, glad your issue got resolved.