[Feature Request] - Allow for and/or Document Steps to resize Physical Volumes

[tylerfontaine]

Due to upstream constraints, pvcs can't be resized in stateful sets, as outlined here: https://github.com/timescale/timescaledb-kubernetes/issues/120

But if there is a workaround to resize the underlying volumes without resizing the claims OR to recreate the statefulset or similar, this would be a great addition to the Admin Guide, allowing for more flexibile deployments via the helm chart.

[pgodfrin]

In addition to increasing a volume size, adding new volumes to either existing mount points, similar to LVM extensions, and new volumes to new mount points, would be very useful.

[drpebcak]

When you talk about a workaround that DOESN'T involve recreating the statefulset, what do you mean?

I think the only real option for this is to manually edit the existing PVCs to resize them, run kubectl delete sts --cascade=false <timescale-sts-name> to delete your statefulset, update the PVC sizes in your helm values, and then trigger a helm upgrade with those new values.

[ss1796]

We run Timescale on EKS. Here are the steps that we follow.

1. Modify the EBS volumes.
2. Update the PVC
3. Delete the statefulset
4. Update the volume definitions in the values.yaml and deploy helm chart to re-create the sts using the existing PVC

Is this a supported way to grow the volume?

[drpebcak]

We run Timescale on EKS. Here are the steps that we follow.

1. Modify the EBS volumes.
2. Update the PVC
3. Delete the statefulset
4. Update the volume definitions in the values.yaml and deploy helm chart to re-create the sts using the existing PVC

Is this a supported way to grow the volume?

Steps 1 and 2 can be collapsed into a single step if you configure the storageclass with allowVolumeExpansion: true. See this blog: https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/

[Nainterceptor]

Any update here or a way to resize without downtime ?

[drpebcak]

@Nainterceptor since k8s 1.15 (technically before, but that's when it was enabled by default) you can resize an in-use pvc, provided your underlying driver supports it: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#resizing-an-in-use-persistentvolumeclaim

As far as I am aware, you still need to step through recreating the statefulset object though, since the volumeClaimTemplates section does not support resizing. If you delete the statefulset with --cascade=false you won't have downtime from this.

If your volume driver DOESN'T support resizing, you still won't have downtime if you are running an HA postgresql cluster. You can remove the pods one by one to avoid that.

[Nainterceptor]

Hello, FYI, we've documented pvc expansion process, and --cascade=ophan is not working in our workflow. Helm instance is created via terraform, and if we delete statefulset without pods, the subsequent terraform apply will fail with message Error: "timescaledb" has no deployed releases I don't know why, but no issue without this cascade option.