Adding volumes to StatefulSet after the initial creation

[leochr]

Kubernetes StatefulSet itself doesn't allow volumes to be added to StatefulSets after their initial creation.

Trying to add a volume, results in this error: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

The same limitation is encountered via operator.

• create StatefulSet by setting:

  spec:
  statefulSet: {}

• try to add a volume to it

  spec.
  statefulSet: 
  storage:
    size: 100Mi
    mountPath: /logs

Tasks:

• [ ] understand the reasons behind StatefulSet not supporting
• [ ] if it's not an anti-pattern to add volume afterwards, support adding it via operator. StatefulSet probably have to be deleted and recreated (meaning there will be some downtime)

[kabicin]

The reason for StatefulSet not supporting modification of other fields (in general):

• Deleting the StatefulSet and pod(s) will not delete bound PVs and it's PVCs. This is to preserve data safety, which is put at a higher precedence than deleting all StatefulSet resources.
• Difficulty determining an update strategy for pods
  • Currently, a non-cascading delete and then re-creation of the StatefulSet will try and reuse it's X orphaned pods still using old claims to PVs, despite any new modifications made to the volume template.
• No guarantee on deletion of all pods when a StatefulSet is deleted

For this specific use case outlined above, PVs and PVCs are not an issue but ensuring deletion of all pods is necessary to ensure recreation with the correct volume from StatefulSetSpec.

Edit: I am reconsidering to not ensure deletion of all pods since this specific use case will never cause a forced rollback state and so we can depend on StatefulSet controller to gracefully delete a pods with statefulSet: {}

[kabicin]

The PR above only implements adding volumes to instances without storage specified, i.e. statefulSet: {}. Given the above findings, do we want to support modifying volumes to RCO instances which already have StatefulSet + volume provisioned? For example, changing the mountPath PV size (which currently updates RC spec but does not actually update the StatefulSet spec until it is manually restarted).

[halim-lee]

I would recommend adding support only for adding volumes for now. Since making changes to StatefulSet volumes also requires deletion of pods, I am not sure if it is safe to make that too available. If we decide to support modifying the existing volumes, we can implement it later.