reactive-tech / kubegres

Kubegres is a Kubernetes operator allowing to deploy one or many clusters of PostgreSql instances and manage databases replication, failover and backup.
https://www.kubegres.io
Apache License 2.0
1.32k stars 74 forks source link

Expand storage of an existing cluster of Postgres #49

Open alex-arica opened 3 years ago

alex-arica commented 3 years ago

Currently, Kubernetes does not allow the support for volume expansion through StatefulSets. And when a PVC is manually updated, it is unable to automatically update a StatefulSet.

To allow to expand the storage of an existing cluster of Postgres, we have to wait on the release of the Kubernetes enhancement kubernetes/enhancements#2842 which aims in addressing this issue by allowing support for volume expansion through StatefulSets.

Once that feature is available in Kubernetes, this feature can be available in Kubegres via this ticket.

ktzsolt commented 2 years ago

In the meantime if anyone tries to resize kubegres pvc-s, here is my step-by-step howto. Might need fine tuning and is as-is so experiment with this in nonprod environments first! Feel free to complement this with further info.

Resize kubegres database

1. disable kubegres operator:

kubectl scale --replicas=0 deployment.apps/kubegres-controller-manager -n kubegres-system

2. check if kubegres controller deployment is scaled down to 0

kubectl get deployment kubegres-controller-manager -n kubegres-system

3. export kubegres statefulset manifest

kubectl get sts kubegres-1 -o yaml > kubegres-1.yaml

4. edit pvc size in exported manifest and replace statefulset with it

kubectl replace --force -f kubegres-1.yaml

5. scale down statefulset

kubectl scale --replicas=0 sts kubegres-1

6. resize PVC

https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/

wait for pvc resize condition message: Waiting for user to (re-)start a pod to finish file system

7. scale backu statefulset

kubectl scale --replicas=1 sts kubegres-1

8. wait for pod(s) to start in statefulset

9. repeat 2-7 with all kubegres statefulsets

10. modify size in your kind: Kubegres manifest in git or wherever you store your manifests

11. enable kubegres operator

kubectl scale --replicas=1 deployment.apps/kubegres-controller-manager -n kubegres-system

jeremybusk commented 2 years ago

Similar. https://serverfault.com/questions/955293/how-to-increase-disk-size-in-a-stateful-set

https://learn.microsoft.com/en-us/azure/azure-arc/data/resize-persistent-volume-claim

Or looks like it just works updating pvc(s) each to size, at least with Azure managed-csi storage class on azure.