search

kubernetes-sigs / aws-load-balancer-controller

A Kubernetes controller for Elastic Load Balancers
https://kubernetes-sigs.github.io/aws-load-balancer-controller/
Apache License 2.0
3.83k stars 1.41k forks source link

Controller Provision a Port Range for the Backend Security Group Rule #3460

Open dinukarajapaksha opened 8 months ago

dinukarajapaksha commented 8 months ago

Describe the bug When the Controller provisioning backend security groups, it adds a port range rule for the backend (EKS Node Security Group) instead for specific ports on the Kubernetes service. We are deploying a Nginx Ingress Controller with the ALBC annotation as the following.

loadBalancerSourceRanges:
  - "10.0.0.0/8"
annotations:
  service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: "true"
  service.beta.kubernetes.io/aws-load-balancer-type: external
  service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip

The service ports for the Nginx service are 80 and 443. Ideally the backend security group should add the rule for only these 2 ports and add the LB security group as the source. Instead the Controller trying to add the security rule for the port range 80-443. Following is log message from the controller

{
  "level": "info",
  "ts": "2023-10-30T11:11:32Z",
  "msg": "authorizing securityGroup ingress",
  "securityGroupID": "<eks-node-group-sg>",
  "permission": [
    {
      "FromPort": 80,
      "IpProtocol": "tcp",
      "IpRanges": null,
      "Ipv6Ranges": null,
      "PrefixListIds": null,
      "ToPort": 443,
      "UserIdGroupPairs": [
        {
          "Description": "elbv2.k8s.aws/targetGroupBinding=shared",
          "GroupId": "<nlb-backend-security-group>",
          "GroupName": null,
          "PeeringStatus": null,
          "UserId": null,
          "VpcId": null,
          "VpcPeeringConnectionId": null
        }
      ]
    }
  ]
}

aws-load-balancer-controller version: 2.6.2

Since this add all the ports between 80-443 it is not a recommended security practice

Steps to reproduce aws-load-balancer-controller version: 2.6.2

Deploy a service that provision a NLB via Controller with the following configuration

loadBalancerSourceRanges:
  - "10.0.0.0/8"
annotations:
  service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: "true"
  service.beta.kubernetes.io/aws-load-balancer-type: external
  service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip

Expected outcome Controller should add a rule for the security group in EKS Node Group for only port 80 and 443 with NLB backend SG as the source. Not a SG rule that contains port 80-443

Environment

Additional Context: I tried different configurations on the documentation but there wasn't a configuration option to handle this. Any help on this matter will be highly appreciated. Thanks in advance

oliviassss commented 7 months ago

@dinukarajapaksha, hi, it's an expected behavior for the controller to add the port range based on the min/max values of the ports since v2.3.0. You mean we should only allow individual ports, instead of allow a port range? Commit: https://github.com/kubernetes-sigs/aws-load-balancer-controller/pull/2236 release note: https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/tag/v2.3.0

dinukarajapaksha commented 7 months ago

Hi @oliviassss, Yes in the security group only the specific individual ports are expected to be open. Otherwise in the SG, unnecessary ports would be allowed which could lead to security violations.

For an example for the port 80 and 443 the AWS Load Balancer Controller opens all the ports from 80 - 443 [80,81,82..]. This include some ports we don't want to open because of security violations on clear text protocols. Such as port 161, 110. I believe this is not the best way to create the SG.

Is there a particular reason why the controller needs to open a port range? If not can we have a better solution on this with individual ports?

james-bjss commented 7 months ago

@dinukarajapaksha, hi, it's an expected behaviour for the controller to add the port range based on the min/max values of the ports since v2.3.0. You mean we should only allow individual ports, instead of allow a port range? Commit: #2236 release note: https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/tag/v2.3.0

We are also observing this behaviour in our environments. When performing security testing of our infrastructure various tools are flagging large port ranges being opened in our SGs. Is it possible to enforce individual rules based on the ports being used? (I realise there is a limitation on the number of rules).

Or will we need to move ports for our applications to narrow the range?

I guess we could provide our own SG and then just set --backend-security-group or disable the functionality?

mcalligator commented 7 months ago

This is definitely an insecure practice; the LBC should create SG ingress rules only for the ports explicitly specified by default, and optionally include a parameter to specify open ports as a range.

For the present, the potential workaround is to specify service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: "false", but that means we have to identify manually which EKS services require which ports open, which is slower and more error-prone.

oliviassss commented 5 months ago

/kind feature

k8s-triage-robot commented 2 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

hobti01 commented 2 months ago

This is a surprising discovery, especially when health checks are on high ports the range becomes large and unexpected. We have cases where ports are open for UDP but should not be open for TCP, but the TCP range now overlaps the UDP-only ports :(

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

RaviVadera commented 4 weeks ago

/remove-lifecycle rotten