search

pulumi / pulumi-kubernetes

A Pulumi resource provider for Kubernetes to manage API resources and workloads in running clusters
https://www.pulumi.com/docs/reference/clouds/kubernetes/
Apache License 2.0
406 stars 116 forks source link

Decide how skipAwait should interact with deletion #1232

Open lblackstone opened 4 years ago

lblackstone commented 4 years ago

Problem description

https://github.com/pulumi/pulumi-kubernetes/blob/d7871675b54727f63f1e2254d06653daf06cc8c8/provider/pkg/await/await.go#L463-L511

For resource types that don't have custom await logic, the provider currently ignores the pulumi.com/skipAwait annotation. If this annotation is set, the provider should immediately consider the resource to be deleted.

Suggestions for a fix

Duplicate the logic here: https://github.com/pulumi/pulumi-kubernetes/blob/d7871675b54727f63f1e2254d06653daf06cc8c8/provider/pkg/await/await.go#L446-L447

blampe commented 3 months ago

Edit: tracking separately https://github.com/pulumi/pulumi-kubernetes/issues/3157

EronWright commented 2 months ago

I wonder how replacement would combine with skipAwait on delete and with SSA. Would the Apply call performed by the replacement be racing against finalization of the original?

blampe commented 2 months ago

After a lot of discussion we've decided to punt on this for the time being. The existing behavior is arguably buggy, so we need to be cautious about expanding the behavior to more resources.

To elaborate on why this is buggy, when we delete resources that respects skipAwait (Deployments, Jobs, StatefulSets, Namespaces, Pods, and ReplicationControllers) we don't wait for the deletion to succeed. In cases where a resource is getting delete-before-replaced, this combines with our SSA upsert behavior (https://github.com/pulumi/pulumi-kubernetes/issues/2998) to create a race condition where the resource could still be getting deleted when we upsert.

The annotation currently provides two functions when deleting relevant resources:

  1. It makes deletion fast, but this is almost always better handled with a background or orphan delete (https://github.com/pulumi/pulumi-kubernetes/issues/2529).
  2. It makes it possible to ignore stuck finalizers, which is generally not recommended but sometimes necessary.

We're currently considering changing the behavior to completely ignore the annotation during deletion. For users who need (1) there's another annotation available. It's unclear how many users actually rely on the annotation for (2) but we assume it's small -- if not, we might need a separate workaround for this case.