search

kubernetes / kops

Kubernetes Operations (kOps) - Production Grade k8s Installation, Upgrades and Management
https://kops.sigs.k8s.io/
Apache License 2.0
15.95k stars 4.65k forks source link

AWS LBC cannot delete Security Groups it creates #13020

Closed johngmyers closed 2 years ago

johngmyers commented 2 years ago

/kind bug

1. What kops version are you running? The command kops version, will display this information.

master branch. Broken code appears to be in release-1.23 branch as well

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

1.21.7

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

Examine the kops-aws-aws-load-balancer-controller e2e test.

Alternatively:

  1. Create a cluster with the AWS Load Balancer controller.
  2. Create an Ingress backed by an ALB.
  3. Delete the ingress

5. What happened after the commands executed?

The ALB controller is unable to delete the "k8s-traffic-" ingress for the ALB due to lack of IAM permissions. This is because the security group is missing the KubernetesCluster tag.

6. What did you expect to happen?

Security group is tagged with the tags that kOps passed AWS LBC through the default-tags flag. LBC is able to delete the security groups it creates.

7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

Introduced by #12920.

Filed upstream as kubernetes-sigs/aws-load-balancer-controller#2424

hakman commented 2 years ago

IMO, anything related to LBC should not block anything. If you want, we can revert the commit in 1.23.

olemarkus commented 2 years ago

We can't ship a broken addon. But we can revert the upgrade. We are still waiting for a few other addons so there is time if upstream moves fast.

hakman commented 2 years ago

We don't want to ship a broken addon, but this is already broken. It doesn't work at all for k8s 1.22+ and we still ship it. We can also make ec2:DeleteSecurityGroup to unconditional action.

johngmyers commented 2 years ago

Not working at all for k8s 1.22+ is not "broken". It still worked for k8s 1.21 and earlier. With this issue it stopped working for k8s 1.21 and earlier.

I don't think we should open up the RBAC. We could change the condition to use their tag, but the controller would still be failing to tag the SG with the admin-provided tags.

hakman commented 2 years ago

I still see it as "broken". I don't mind reverting the update. I don't mind even removing it from the kOps addons.

justinsb commented 2 years ago

@olemarkus mentioned that the new version of LBC has been merged so can we close this?

johngmyers commented 2 years ago

This can probably be closed, but I can't test it right now.