Open shreyas-s-rao opened 1 year ago
/assign
@shreyas-s-rao as you have already found out that just by changing the storageCapacity
a statefulset cannot be updated. If one tries to do that the following error will be returned:
The StatefulSet "web" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
For a multi-node etcd if you wish to avoid down time and also change the storageCapacity
to a lower capacity then there is one way to do it:
RollingUpdate
to OnDelete
for the statefulset.--cascade=orphan
so that it does not delete the pods.storageCapacity
(25Gi) and recreate the statefulset.Repeat this for all the pods and this way you would be able to update all etc statefulset replicas while maintaining quorum. Since you have control over the update process, you can wait till the new etcd member has been promoted to a member.
We anyways should change the update strategy from RollingUpdate
to OnDelete
to ensure that quorum is always maintained which is not possible to ensure when using RollingUpdate
How to categorize this issue?
/area backup /area storage /area cost /kind enhancement /platform aws
What would you like to be added: Update ETCD volume (GP3) to use lower storage capacity.
Why is this needed: The current storage capacity of
80Gi
was used for GP2 volumes for ETCD data storage, since the IOPS for GP2 volumes is determined by the volume size. Hence,80Gi
was set as the volume size in order to achieve optimum IOPS for ETCD, even though the ETCD DB and additional operations do not utilise the entire 80GB of storage. With the move toGP3
volumes, this is no longer necessary - the storage capacity can be reduced to a lower value sufficient enough to host the ETCD DB and allow additional operations on it (such as defragmentation, restoration, etc).