search

zalando / postgres-operator

Postgres operator creates and manages PostgreSQL clusters running in Kubernetes
https://postgres-operator.readthedocs.io/
MIT License
4.37k stars 980 forks source link

add cluster field for PVCs #2785

Closed FxKu closed 3 weeks ago

FxKu commented 1 month ago

PVCs are the only resource which is not covered by OwnerReferences. We have observed that on cluster deletion operator can report that all PVCs have already been deleted, although they still exist. We believe that there might be a race condition with cluster fields such as Name and Namespace getting unset after cascaded deletion via OwnerReferences resulting in an empty list when fetching PVCs to delete (labelSet function using c.Name for filtering). PVC management by StatefulSet is set to Retain in our case.

This PR present an idea which we use for other resources as well: Storing resource information in a cluster field and loop over this map on deletion. However, unlike resources like Secrets and Services, PVCs are created by K8s via StatefulSet template. So we rely on a syncVolumeClaims call to have the PVCs info stored in the cluster struct. If this sync does not happen for some reason, the PVCs will not get deleted. I don't think an extra call of listPersisntentVolumeClaims is needed as c.Name would mostly likely be empty as well in this scenario.

I found that syncVolumes is not called in Create() event. Adding it there would greatly reduce the chance that PVCs are not synced before a potential deletion.

hughcapet commented 3 weeks ago

👍

FxKu commented 3 weeks ago

:+1: