search

Stackdriver / stackdriver-prometheus-sidecar

A sidecar for the Prometheus server that can send metrics to Stackdriver.
https://cloud.google.com/monitoring/kubernetes-engine/prometheus
Apache License 2.0
120 stars 43 forks source link

Delete the replica sets from the old deployment when patching the deployment. #243

Closed qingling128 closed 4 years ago

qingling128 commented 4 years ago

Work around the issue when PVC is still being held by the old pods from the old deployment's replica set when patching the deployment:

  Warning  FailedAttachVolume  7m8s                attachdetach-controller                                       Multi-Attach error for volume "pvc-dc041109-cb02-11ea-82bf-42010a8001b2" Volume is already used by pod(s) prometheus-server-77bc48c5cb-qh5gp
  Warning  FailedMount         34s (x3 over 5m7s)  kubelet, gke-lingshi-prom-sidecar-default-pool-e7c128e9-btnb  Unable to mount volumes for pod "prometheus-server-6df94f65d5-wjbkf_prometheus(e23e0c7c-cb02-11ea-82bf-42010a8001b2)": timeout expired waiting for volumes to attach or mount for pod "prometheus"/"prometheus-server-6df94f65d5-wjbkf". list of unmounted
volumes=[storage-volume]. list of unattached volumes=[config-volume storage-volume prometheus-server-token-8s9mr]

Another option is to set .spec.revisionHistoryLimit to 0 for this deployment so that orphaned replica set gets deleted right away, but I feel like that is a bit too aggressive. Open to other suggestions as well.

qingling128 commented 4 years ago

Added the --clean-up-orphaned-replica-sets flag. Please take a look.