Kubernetes receives a request to downscale a statefulset by X hosts.
The prepare-downscale admission webhook attempts to prepare X pods for shutdown by sending an HTTP POST to their handler identified by the grafana.com/prepare-downscale-http-path and -port annotations.
At least one of these requests fails. The admission webhook returns an error to Kubernetes, so the downscale is not approved.
💥 But some hosts may have been prepared for downscale. 💥
This PR adds cleanup logic to issue DELETE requests on all involved pods if any of the POSTs failed. Notes:
DELETE calls are attempted once.
DELETE failures are logged but otherwise ignored.
For simplicity, we'll invoke DELETE on all of the pods involved in the scaledown operation, not just ones that received a POST.
This doesn't fix the similar issue where replica count changing from 10->9->10 leaves that one pod prepared for shutdown. (But that's in the works.)
This addresses a bug in rollout-operator where:
X
hosts.X
pods for shutdown by sending an HTTPPOST
to their handler identified by thegrafana.com/prepare-downscale-http-path
and-port
annotations.This PR adds cleanup logic to issue
DELETE
requests on all involved pods if any of thePOST
s failed. Notes:DELETE
calls are attempted once.DELETE
failures are logged but otherwise ignored.DELETE
on all of the pods involved in the scaledown operation, not just ones that received a POST.This doesn't fix the similar issue where replica count changing from 10->9->10 leaves that one pod prepared for shutdown. (But that's in the works.)