[EKS] [request]: Allow cloud provider config. setting DisableSecurityGroupIngress

[kh34]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request Could EKS allow us to specify cloud provider parameters such as DisableSecurityGroupIngress and ElbSecurityGroup?

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

My company is trying to migrate from self-managed Kubernetes to EKS. Our current setup consists of several hundred type LoadBalancer Service resources. Each developer manages their own ingress to the cluster in this way. When trying to migrate this workload, we ran into an error from reaching the maximum inbound or outbound rules per security group. I am aware of:

https://docs.aws.amazon.com/vpc/latest/userguide/amazon-vpc-limits.html#vpc-limits-security-groups

and the recent updates: https://aws.amazon.com/premiumsupport/knowledge-center/increase-security-group-rule-limit/

But even these possible increases are not enough to "lift-and-shift" our workload. It's not possible for us to undertake a significant refactor in order to migrate to EKS at this time.

Are you currently working around this issue? How are you currently solving this problem?

We currently manage security group creation manually. We set the cloud provider config. file with the options:

[Global]
DisableSecurityGroupIngress = true
ElbSecurityGroup = sg-XXX   <== This is a group with 0 ingress/egress rules

And pass --cloud-config= to our control plane components and kubelets. Then we manage both the Node security group/ELB security group outside of Kubernetes. A SG is created for the ELB and attached to the node SG's ingress rules. Then we add the following annotation to the Service manifest to attach it:

service.beta.kubernetes.io/aws-load-balancer-extra-security-groups

This solution gives us the control we need to stay within the security group limitations.

Refer to: https://github.com/kubernetes/cloud-provider-aws/commit/dbc66dc675eee8b53c399a5c300122e37d3fade0 https://forums.aws.amazon.com/thread.jspa?messageID=896987

Additional context

I am aware of other possible workarounds, such as:

• Sharing a single AWS load balancer amongst multiple applications and deploying an ingress controller. We have concerns over this from a blast radius perspective - one poorly behaving application can trigger outages for others, refer to https://aws.amazon.com/premiumsupport/knowledge-center/elb-capacity-troubleshooting/

Attachments If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

[jicowan]

Fixed with https://github.com/kubernetes/legacy-cloud-providers/blob/e68b55f48d8efc43af007916afc0d560838e9c7f/aws/aws.go#L153-L156 This update has been backported to EKS 1.14 and 1.15.

[astrived]

@kh34 given that the ServiceAnnotationLoadBalancerSecurityGroups = "service.beta.kubernetes.io/aws-load-balancer-security-groups" gives you the functionality that you need i.e. using this annotation your can provide the security group you want do you see a need for any change ?

[kh34]

In short, no need for a change, and this issue can probably be closed. Enforcing the use of this new annotation in an enterprise or in a multi-tenant cluster is probably out of scope for this discussion. We'll be looking into developing our own custom operator or MutatingWebhook for that.

[dee-kryvenko]

https://github.com/kubernetes/legacy-cloud-providers/blob/e68b55f48d8efc43af007916afc0d560838e9c7f/aws/aws.go#L153-L156 is not suffice - EKS still tries to find workers SG and modify it. And if I don't have workers security groups tagged, it gives

Error creating load balancer (will retry): failed to ensure load balancer for service default/svc: Multiple untagged security groups found for instance i-******; ensure the k8s security group is tagged

Our security requirements does not allow SGs to be modified by EKS, hence either DisableSecurityGroupIngress or something else disabling that https://github.com/kubernetes/legacy-cloud-providers/blob/e68b55f48d8efc43af007916afc0d560838e9c7f/aws/aws.go#L3975-L3979 behavior is required. K8s version 1.14.

[alfredkrohmer]

We have the exact same problem as described by @dee-kryvenko.

[alindeman]

@dee-kryvenko Did you find any workaround or are you still blocked on this? I'd also prefer to disable security group modifications by the EKS/AWS cloud provider, and instead manage them in my own configuration.

[dee-kryvenko]

I was not able to find any workaround. I ended up creating an empty tagged SG with a meaningless rule 127.0.0.1/32 along with the cluster - my security team were not happy about that.

[lstoll]

@astrived is it possible to re-open this one? I don't think the issue is completely resolved. Specifying the annotation gives you control over the SG the Load Balancer uses, but does not stop the cloud provider modifying the node security groups.

[astrived]

@lstoll yes I am tracking this, will keep it open.

[gazal-k]

We're facing a similar issue. We don't have a lot of loadbalancer services, but we just have a lot of loadBalancerSourceRanges in our ingress service. We've requested for the limit to be increased to the maximum possible which will help us for now.

For each entry in loadBalancerSourceRanges EKS creates the following inbound rules:

• MTU
• one each for each port

So the standard NLB service with both port 80 & 443, that's 3 inbound rules per CIDR. Then there's the health rules per port and the 2 standard rules for communication with the control plane.

So adding a service that uses an NLB without any loadBalancerSourceRanges already would use 2 (control plane comms) + 2 (health check for each port) + 2 (default 0.0.0.0/0 rules on each port) + 1 (mtu) = 7 inbound rules.

This leaves room for 18 loadBalancerSourceRanges to reach the default limit of 60 on the SecurityGroup. If the limit is increased to the maximum of 100, it allows for 32 loadBalancerSourceRanges.

We're also thinking of removing port 80 from the mix to shave off some of the inbound rule requirement.

But I wonder if EKS can do this in a more flexible way? Perhaps create an SG each for mtu and each traffic port ? This still has the potential of breaching the "Security groups for each network interface" limit. Just throwing the idea out there

[otterley]

Hi everyone,

The introduction of Security Groups for Pods, along with the AWS Load Balancer ingress controller v2, may provide a workable solution to the issue of Security Group rule starvation.

When you use the Load Balancer Controller v2 solution instead of the in-tree Load Balancer controller provided by upstream, it is capable of using IP-based targets for your LB Target Groups instead of instances. In addition, the controller will recognize when your Pod is using a dedicated Security Group. If so, it will add the necessary rules only to the Pod's Security Group, and not to the cluster Security Group.

I encourage everyone to check out this solution and let us know whether it addresses their issues.

[shivani-hotstar]

Hi, I am also facing the similar issue. Here AWS load balancer ingress controller also not useful . Because i m using v2 version of aws load balancer ingress controller only . The controller helps resolving this issue for ingress objects but not for service type Load balancer objects .

When i am creating a kubernetes load balancer type service . It adds the load balancer security group to worker node group security group as an inbound rule . Which i don’t want.

The service config i am using .

`kind: Service apiVersion: v1 metadata: name: hello-world namespace: test annotations: service.beta.kubernetes.io/xyz-load-balancer-healthcheck-request: / spec: type: LoadBalancer loadBalancerSourceRanges:

127.0.0.1/32 selector: app: hello-world ports: name: http protocol: TCP port: 80 targetPort: web`

Aws load balancer version i am using . 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-load-balancer-controller:v2.0.1 My worker sg is already opened for all-traffic from vpc-cidr range. so no need of these entries here.

Reason why i don’t want load balancer sg entries to worker node group sg . Because of sg rule limit .

Same config is working fine under kops (1.17) but not in eks(1.18)

[dickeyf]

I also have the SecurityGroup entry limit issue in EKS. I didn't have the issue with Rancher or Openshift which allowed me to change the "DisableSecurityGroupIngress" cloud setting. EKS doens't and the annotations doesn't allow the code path to be disabled either. So I am stuck with the limit with EKS currently.

I tried aws lb controller, using nlb-ip. This removed the limit, because all my services works over the same set of ports. Sending traffic to the ports directly to a pod ID (instead of nodeports) solved my issues because the number rules of required doesn't grow as I am adding new services in K8S. So hopefully this can help some other people.

However, using NLB-IP is not optimal for my use case. I send traffic to an Active/Standby system, so, this means the NLB will have two IP targets, one in a different AZ. I also use HTTP HC, so the LB is aware which pod is active. This means the LB will have one healthy target in AZ1, and an unhealthy in AZ2. Now, say I delete the active pod to test a failover. I see this:

1. Target 1 is now deregistering (Takes a long time. This is okay, as long the LB stops sending traffic there).
2. Target 2 takes around 30-40 seconds to becomes healthy (Using 10s interval with 2 consecutive attempts). So far so good
3. However, traffic still doesn't make it to the health target. It takes another minutes (At least!) before the NLB starts forwarding there. I find the NLB isn't working fine somewhere... shouldn't take that long for a LB to start forwarding to a newly healthy target.
4. At some point a new target appears, replacing the one I deleted. It takes around 5 minutes for it to be unhealthy (This is not a problem however... if only LB could forward traffic as soon as a target registers as healthy).

I tried enabling cross-zones, in case this was due to DNS TTL. But it doesn't help. Both frontend IPs takes a long time to start forwarding traffic to the new healthy pod.

For this reason, I'd rather stick to the NLB -> NodePort model as K8S takes care of routing traffic to healthy pod, this is allows way faster failovers (15 seconds or so).

I believe that for the aws-lb-controller to truely solve the problem here, it would need to implement both "nlb" and "nlb-ip" thus completely replacing the in-tree driver. So the controller would be able to do NLB -> NodePort, and exposes the cloud settings settings "ie. DisableSecurityGroupIngress" via ConfigMaps (Which could be configured by helm parameters at lb-controller deployment time).

[recollir]

Any update on this issue? As we are also affected by this and it would require a lot of workarounds to move EKS from kops managed clusters.

[luanhdeandrade]

I am here to reinforce the importance of having this feature. The absence of it is causing us a lot of headaches