Consider allowing for delete of underlying PVC when StatefulSet is deleted

[hausdorff]

The official client-go delete semantics for StatefulSet do not cause the underlying PersistentVolumeClaim to be deleted. This catches nearly all new users off-guard, and is only evident when they try to create more PVs, and provisioning fails because all the storage is taken up by these orphaned resources.

Some ideas on how to fix:

• Pulumi engine schedules a manual delete of the underlying PVC any time a StatefulSet is deleted
• Implement some kind of StrictStatefulSet, where this is the default behavior (similar to our plans for, e.g., StrictDeployment)
• Extend StatefulSet base class to have (e.g.) ReclaimingStatefulSet, which allows users to specify a ReclaimPolicy, which is manually propagated to the underlying PVC.
• Add post-delete hooks, similar to the .apply function.

Each of these has positives and negatives. I'm leaning toward the second one, as it means that we gracefully degrade to the Kubernetes delete semantics, and gives people the ability to opt into less permissive semantics. The third one is not bad, but it effectively means writing our own StatefulSet API, which kind of clashes with our 1:1 model of the Kubernetes API.

[geekflyer]

I prefer 3. Might be worthy to think about naming it something other than "ReclaimPolicy" because that's strictly speaking just an attribute on a PV and not a StatefulSet or PVC and using the same name might confuse people, but I might be wrong.

[hausdorff]

Since the *-x packages are actually being worked on, I actually think (and feel comfortable proposing) that we keep the behavior in the Kubernetes core SDK identical to that of Kubernetes upstream, and then expose special knobs along the lines of (2) or (3) that users can do to modify this behavior. So, I'm going to untag this from M21, tag it with the kubernetes/x tag, and make a note to talk about it at the next kube-x sync meeting.

This proposal comes after months of background reflection, as well as soliciting opinions from community, Pulumi kube team, and so on, so I'm reasonably confident this is the right move. If someone doesn't think that's true, let's talk about it, but for now I'm going to proactively move the issue out of the milestone.

[hausdorff]

cc @lblackstone @metral @lukehoban

[lukehoban]

This seems like something we would do additively - without requiring changes to existing Pulumi. Each of (2), (3) and (4) feel like things we can explore on top of the existing pulumi-kubernetes foundation.

[lbialy]

Hi, did you figure out how to deal with this and to allow graceful clean up when a statefulset gets deleted? I hit this while working on pulumi-scala over the weekend and it's an surprising problem to say the least least!