pulumi / pulumi-kubernetes

A Pulumi resource provider for Kubernetes to manage API resources and workloads in running clusters
https://www.pulumi.com/docs/reference/clouds/kubernetes/
Apache License 2.0
407 stars 117 forks source link

helm.v4.Chart: make helm hooks optional (instead of force disabled) #3284

Open awoimbee opened 2 weeks ago

awoimbee commented 2 weeks ago

Hello!

Issue details

helm.v3.Chart ignored the helm.sh/hook* annotations but still deployed the resources. The helm.v4.Chart resource skips the hooks. This is mentioned in the blog but not in the docs BTW.

I think users should have a way to make v4.Chart deploy hooked resources like v3 did, because it's useful and becasue it makes migrating from v3 to v4 less painful.

EronWright commented 1 week ago

I think the issue is that hook resources have special lifecycle/ordering requirements. Basically would they work as expected?

awoimbee commented 1 week ago

Yep, I have multiple charts that stopped working when moving from v3 to v4:

The issue I have with just removing the hooks is that I can't get the templated yaml out of pulumi. If I want to remove a hook, I can do it myself, using something like (this v3 transform):

export const ignoreResourcesHelmTransformation = (...resources: string[]) => {
  const stack = pulumi.getStack();
  return (o: any, opts: pulumi.CustomResourceOptions) => {
    const resourceName = `${o.apiVersion}:${o.kind}:${o.metadata?.namespace || ""}:${o.metadata?.name || ""}`;
    if (resources.includes(resourceName)) {
      console.log(`Skip: ${resourceName}`);
      const dir = `../ignored-resources-${stack}`;
      if (!fsSync.existsSync(dir)) {
        fsSync.mkdirSync(dir);
      }
      fsSync.writeFileSync(
        `${dir}/${resourceName.replaceAll("/", "-")}.yaml`,
        yaml.dump(o)
      );
      for (const key of Object.keys(o)) {
        delete o[key];
      }
      o.apiVersion = "v1";
      o.kind = "List";
      o.items = [];
    }
  };
};
EronWright commented 1 week ago

(here's an outline of the technical challenge)

The Helm documentation outlines the various hook behaviors that Pulumi would be expected to emulate: https://helm.sh/docs/topics/charts_hooks/#the-available-hooks

In v3, Pulumi simply treated the hooks as normal resources, which can lead to incorrect behavior.

As we can see, hooks generally affect the order of operations, and also vary based on whether the chart is being installed, upgraded, or deleted. The Pulumi engine provides the dependsOn option to control order-of-operations between resources, which is likely sufficient to support the pre/post aspect. It is less clear how to support "upgrade" hooks, since the Chart/v4 resource is stateless, it cannot distinguish between install and upgrade. Also unclear how to support "delete" hooks, since the Chart/v4 resource is a component resource and there's no opportunity to "create" a hook resource during component deletion.