Allow to pass `bootstrapSelfManagedAddons` argument to cluster

pulumi / pulumi-eks

A Pulumi component for easily creating and managing an Amazon EKS Cluster

https://www.pulumi.com/registry/packages/eks/

Apache License 2.0

171 stars 81 forks source link

Allow to pass `bootstrapSelfManagedAddons` argument to cluster #1475

Open ahanoff opened 1 week ago

ahanoff commented 1 week ago

Hello!

Vote on this issue by adding a 👍 reaction
If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

Currently EKS cluster sets by default bootstrapSelfManagedAddons as true which can not be overridden. That setting creates 3 addons: vpc-cni, coredns, and kube-proxy

As a workaround users can use the defaultAddonsToRemove input property with ["vpc-cni", "coredns", and "kube-proxy"] to remove all addons on cluster creation.

Affected area/feature

Prevents from creating clean cluster without any addon which is beneficial if you use custom CNI like Cilium

t0yv0 commented 1 week ago

Hi @ahanoff thanks for the issue. Have you tried the defaultAddonsToRemove property?

https://www.pulumi.com/registry/packages/eks/api-docs/cluster/#defaultaddonstoremove_nodejs

I believe if you populate that property with a list of "vpc-cni", "coredns", "kube-proxy" then they should not be auto-created anymore.

ahanoff commented 6 days ago

Have you tried the defaultAddonsToRemove property?

hi @t0yv0, thanks! Yes, that's what I currently set while waiting for this issue resolution

List of addons to remove upon creation. Any addon listed will be "adopted" and then removed

Actually I'm confused with two things:

defaultAddonsToRemove comment above as I'm not sure what adopted means in this case. If it means it will be installed first and then removed, would it be a case when AWS VPC CNI leaves some iptables rules that needs to be cleared?
having two options (bootstrapSelfManagedAddons and defaultAddonsToRemove) to do same things?

Anyway I'm going test your suggestion this week and get back with feedback

t0yv0 commented 6 days ago

Great feedback, let me double-check and see if we can clarify the docstrings.

flostadler commented 5 days ago

Hey @ahanoff, defaultAddonsToRemove wouldn't leave any ip tables rules behind because the adopt->delete procedure is done before there's any nodes in the cluster. In detail, we're creating an EKS Addon for every addon you mention in defaultAddonsToRemove. This adopts the existing resources on the cluster into Pulumi's management. Finally, we're deleting those addons again, which cleans up the underlying resources.

But I agree with you that having two options to achieve the same goal is not ideal! pulumi-eks inherits this option from the pulumi-aws provider that manages the underlying cluster resource. defaultAddonsToRemove was added to that provider before AWS added the bootstrapSelfManagedAddons feature. I created an issue in pulumi-aws to track unifying this situation: https://github.com/pulumi/pulumi-aws/issues/4755

ahanoff commented 4 days ago

have you tried the defaultAddonsToRemove property?

hi @t0yv0, @flostadler, I've tested this and would like to confirm that addons are removed and Cilium CNI installed passing cilium connectivity test successfully

flostadler commented 4 days ago

Awesome, thanks @ahanoff. I'll keep the issue open to track this enhancement once we tackled https://github.com/pulumi/pulumi-aws/issues/4755.

I also updated the description with the defaultAddonsToRemove workaround