Add option keep-data when deleting postgres cluster currently not available in 5.1.0-0 postgres-operator

[boers001]

Use Case

We have a multi tenant kubernetes cluster where customers can create postgres-clusters using the postgres-operator-v5 that has been installed. Customers sometimes want the ability to delete the postgrescluster object but want to keep the configured pvc for the database instances and repositories. In version 4.7.x of the postgres-operator you could use the pgo utility to delete the cluster and use the option --keep-data, so you could delete the cluster but the pvc's where not deleted and could later on be reattached when creating a new database cluster.

In the documentation of postgres-operator-v5 in the Migrate to PGO v5 chapter the following is mentioned: "It is recommended to set the reclaim policy for any PV’s bound to existing PVC’s to Retain to ensure data is retained in the event a PVC is accidentally deleted during the upgrade.""

This could be an alternative way but this is not a method a user on the our multi-tenant cluster can use as for setting the reclaim policy you need admin rights which normal users don't have.

It is possible add an option --keep-data that lets the operator know not to delete the configured pvc in the postgrescluster definition so that data within the pvc's can be reused when recreating the database cluster.

If you have any alternative for the above mentioned use case I would appreciate suggestions how to implement this.

Environment

Please provide the following details:

• Platform: OpenShift`,
• Platform Version: 4.10.21
• PGO Image Tag:ubi8-5.1.0-0
• Postgres Version 14
• Storage: kubernetes.io/cinder

[benjaminjb]

Hello, to make sure I'm hearing you right: in this multi-tenant environment, users don't have permissions (and so cannot change PV reclaim policy); so would it possible to set up PGO to handle that?

A few questions/notes: 1) in this multi-tenant environment, would PGO be granted permissions to do that? That seems like something that some admins wouldn't be comfortable with.

2) As for one-off alternatives, you're right, I'm not sure there's a way to do that that doesn't rely on certain cluster-wide privileges to alter PVs. Though I suppose taking backups to a non-PV location might work; that is, if a cluster is being backed up to, say, AWS S3, then those backups can be reused after the cluster is deleted.

3) But if this is a widespread issue that you foresee affecting future clusters, what about defining/using a storage class with the reclaimPolicy of Retain for your PVs? That would then preserve the PV after the cluster was deleted (and then the issue would be "how do we delete PVs with the status Released?").

[benjaminjb]

Not sure if this would fit your use-case, but someone on the team noted that the shutdown option on the spec will shutdown the pods, but leave the volumes. Would that be a potential solution for your use-case?

Also, you'll notice I changed some labels around--we've added this feature to the backlog.

[ValClarkson]

We have added this feature request to our backlog for future development

[boers001]

Hi, for now we are using the third option by changing the retain option of the persistent volume. But this is not the most effective way as in our multitenant cluster with different customers using postgres-operator the customers cannot change the reclaim policy because you can only do this with administrative permissions. We currently use the steps in this blog https://www.crunchydata.com/blog/keep-storage-and-backups-when-dropping-cluster-with-pgo. But this is an labour intensive process and can only be done with good communication between customer and administrator. We prefer that the customer is in control and can easily create a new cluster and keep the existing data in the current pvc. The previous version of postgres-operator 4.7.x you could use the pgo client and delete the cluster with the flag --keep-data that prevented the pvc from being deleted. you could refer to the existing pvc when creating new cluster and you had the same data in the newly created database cluster without having any interaction with the cluster administrator. Would appreciate if this functionality would be available in the postgres-operator v5 in the near future.