new annotation: `deletionPropagationPolicy`

[EronWright]

Proposed changes

Introduces a new Pulumi annotation to set the deletion propagation policy, e.g. to support non-cascading delete on StatefulSet to preserve pods during replacement (see walkthrough).

Note that the policy annotation must be set on the old resource before deleting or replacing it; setting it on the replacement or on the live object is ineffective.

Related issues (optional)

Closes #1831

Example

This example serves to show how the 'orphan' propagation policy allows for non-disruptive replacement of a StatefulSet, e.g. by touching the volumeClaimTemplates.

name: issue-1831
runtime: yaml
description: A StatefulSet to demonstrate the `pulumi.com/deletionPropagationPolicy` annotation.
config:
  # disable SSA for this demonstration
  kubernetes:enableServerSideApply: false
resources:
  nginx:
    type: kubernetes:apps/v1:StatefulSet
    properties:
      metadata:
        name: nginx
        annotations:
          pulumi.com/deletionPropagationPolicy: "orphan"
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: nginx
        serviceName: nginx
        template:
          metadata:
            labels:
              app: nginx
          spec:
            containers:
            - image: nginx:1.19.9
              name: nginx
              ports:
              - containerPort: 80
                name: web
        volumeClaimTemplates:
        - metadata:
            name: nginx
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi

Following the initial deployment, we have these objects:

❯ kubectl get statefulset,pod,pvc -ocustom-columns='KIND:.kind,NAME:.metadata.name,UID:.metadata.uid'
KIND                    NAME            UID
StatefulSet             nginx           b1aa144d-3f16-448e-8e15-02d2c2d4b61a
Pod                     nginx-0         c00d97cc-39d7-4a95-b839-0910f911dbca
PersistentVolumeClaim   nginx-nginx-0   2c624ff9-e856-4d2d-bfaf-527c6b770bc7

To provoke a replacement, we change the PVC template:

              requests:
-                storage: 1Gi
+                storage: 2Gi

Let's also increase the replicas:

      spec:
-        replicas: 1
+        replicas: 2

And deploy:

❯ pulumi up -f
Updating (dev)

     Type                               Name            Status            Info
     pulumi:pulumi:Stack                issue-1831-dev                    4 warnings; 2 messages
 +-  └─ kubernetes:apps/v1:StatefulSet  nginx           replaced (2s)     [diff: ~spec]

Resources:
    +-1 replaced
    1 unchanged

Looking again at the objects:

❯ kubectl get statefulset,pod,pvc -ocustom-columns='KIND:.kind,NAME:.metadata.name,UID:.metadata.uid'
KIND                    NAME            UID
StatefulSet             nginx           135d9142-460c-4f64-82a6-8ce23427f52b
Pod                     nginx-0         c00d97cc-39d7-4a95-b839-0910f911dbca
Pod                     nginx-1         8c80932f-7051-4fc7-baeb-00dae4b07b64
PersistentVolumeClaim   nginx-nginx-0   2c624ff9-e856-4d2d-bfaf-527c6b770bc7
PersistentVolumeClaim   nginx-nginx-1   e4b4fd18-28b2-454b-8c6b-9fa06435d3d6

We see the expected result: the StatefulSet was replaced, the existing pod was adopted, and a new pod was added w/ a PVC.

In more detail, the StatefulSet controller uses the selector to identify existing pods, then chooses to delete or adopt based on suitability. Note the updated owner reference on nginx-0:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-0
  uid: c00d97cc-39d7-4a95-b839-0910f911dbca
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: nginx
    uid: 135d9142-460c-4f64-82a6-8ce23427f52b

To demonstrate how the StatefulSet controller might choose to delete the existing pod rather than adopting it, let's change the image rather than the replicas:

            containers:
-            - image: nginx:1.19.9
+            - image: nginx:1.19.10

              requests:
-                storage: 2Gi
+                storage: 3Gi

We deploy again and see that all pods were replaced.

KIND                    NAME            UID
StatefulSet             nginx           ead53943-abc9-47c8-9393-326f845c7f42
Pod                     nginx-0         74752b8c-3979-478b-9be4-ff3ca1b0aa6f
Pod                     nginx-1         b6c2f0f6-f5ff-4e04-a1da-66966b8d697c
PersistentVolumeClaim   nginx-nginx-0   2c624ff9-e856-4d2d-bfaf-527c6b770bc7
PersistentVolumeClaim   nginx-nginx-1   e4b4fd18-28b2-454b-8c6b-9fa06435d3d6

Note that PVC nginx-nginx-0 was not replaced and still has storage: 1Gi.

[github-actions[bot]]

Does the PR have any schema changes?

Looking good! No breaking changes found. No new resources/functions.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 53.84615% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 32.34%. Comparing base (098bd43) to head (4138b4d).

Files	Patch %	Lines
provider/pkg/await/await.go	0.00%	4 Missing :warning:
provider/pkg/metadata/overrides.go	77.77%	2 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #3011 +/- ## ========================================== + Coverage 32.29% 32.34% +0.05% ========================================== Files 69 69 Lines 8947 8954 +7 ========================================== + Hits 2889 2896 +7 Misses 5791 5791 Partials 267 267 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.