Don't make services wait for pods by default

[davidthor]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

In my first experience with the Kubernetes provider, I ran into an issue of long-running jobs when creating services because it was "Finding pods to direct traffic to". I was creating my services before my deployment/pods so this was innevitable. I pretty quickly found the workaround of using the annotation, pulumi.com/skipAwait, but I'm not a fan of it for a couple reasons:

1. The annotations are written to the k8s resource (as they should be), and k8s annotations are usually used to help operators and other k8s tools to collect metadata and tweak their behavior. This annotation is NOT intended to be scraped by kubernetes and is only used to tell Pulumi how to behave for a resource IT created.
2. There is intentionally a loose relationship between services and their underlying deployments/pods. Having zero pods backed by a service is a valid service definition in k8s on purpose, and there are many cases where pods need access to a dynamic property of the services (see https://github.com/pulumi/pulumi-kubernetes/issues/1648 and https://github.com/pulumi/pulumi-kubernetes/issues/1875). Even the K8s docs point to cases where it is necessary to create services before deployments/pods in order to access environment variables:

Note there's no mention of your Service. This is because you created the replicas before the Service. Another disadvantage of doing this is that the scheduler might put both Pods on the same machine, which will take your entire Service down if it dies. We can do this the right way by killing the 2 Pods and waiting for the Deployment to recreate them. This time around the Service exists before the replicas. This will give you scheduler-level Service spreading of your Pods (provided all your nodes have equal capacity), as well as the right environment variables:

There are really two different feature requests here as a result:

1. Don't make services wait for pods by default, thus preventing the need for the skipAwait annotation.
2. Find something other than k8s annotations to use for informing Pulumi of how to behave for resources that it creates. Pulumi configuration shouldn't be written to the k8s control plane.

Affected area/feature

Pulumi kubernetes provider, specifically the Service resource.

[rquitales]

@davidthor Thanks for the detailed write up and constructive suggestions. I think there is basis for most of what you brought up. I'll discuss this with the rest of the team to see what improvements we could do for working with K8s service resources.

Thanks for using our provider and providing valuable user experience feedback!

[EronWright]

It seems concerning that the default behavior would be changed. I can imagine cases where waiting is important, e.g. if a subsequent resource expects to use the service endpoint.

I agree that the Pulumi annotations shouldn't be written to the object, they're like meta-annotations. It isn't unreasonable that annotations are used in the Pulumi programming model (as opposed to, say, resource options), because they are usable in a variety of contexts (such as in a Helm Release or a ConfigFile resource), but shouldn't be rendered.