search

kubernetes-sigs / aws-load-balancer-controller

A Kubernetes controller for Elastic Load Balancers
https://kubernetes-sigs.github.io/aws-load-balancer-controller/
Apache License 2.0
3.82k stars 1.41k forks source link

Issue with Reusing Pre-existing NLB Created via Terraform in Kubernetes Service #3576

Open wbar opened 4 months ago

wbar commented 4 months ago

Description:

I am encountering an issue when trying to integrate a Kubernetes Service with a pre-existing Network Load Balancer (NLB) that was created via Terraform. Despite correctly tagging the NLB and configuring the Service with the necessary annotations, I receive an error indicating a conflict due to the NLB having "the same name but with different settings". This seems related to the management of Security Groups by the AWS Load Balancer Controller, which has been introduced in a recent feature update for Network Load Balancers.

Environment:

Steps to Reproduce:

  1. Create an NLB using Terraform with specific tags: 
    ingress.k8s.aws/stack   XXXXXXXX
elbv2.k8s.aws/cluster
ingress.k8s.aws/resource
  2. Configure a Kubernetes Service with annotations to use the pre-created NLB and specify the load balancer settings, 
    service.beta.kubernetes.io/aws-load-balancer-alpn-policy: HTTP2Preferred
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-ip-address-type: ipv4
service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: 'false'
service.beta.kubernetes.io/aws-load-balancer-name: XXXXXXX
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:ZZZZZZZ
service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: SOME_ATTRS 
service.beta.kubernetes.io/aws-load-balancer-type: external
  3. Attempt to deploy the Service in a new environment.

Expected Result: The Kubernetes Service should successfully associate with the pre-created NLB without any conflicts regarding the load balancer's name or settings.

Actual Result: Received an error message:

A load balancer with the same name 'XXXXXXX' exists, but with different settings.

This suggests an issue with how the AWS Load Balancer Controller handles existing NLBs, particularly regarding Security Groups settings.

Additional Information:

wbar commented 4 months ago

Everything works on version 1.5.5

wweiwei-li commented 4 months ago

We do not support using existing NLB for service now. But we are have a feature soon, You can track it here : https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/228

TargetGroupBlinding supports Existing ALB/NLB, can you take a look to see if it helps with your case: https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.7/guide/targetgroupbinding/targetgroupbinding/

wbar commented 4 months ago

Thanks for a response.

You have been supporting this till version 1.5.5.

Now, when AWS introduced Security Groups for NLB and you changed you Controller to use this feature, you are totally ignoring:

service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: 'false'

Even adding this:

service.beta.kubernetes.io/aws-load-balancer-security-groups: ''

wont help because AWS Controller is building model with security groups:

// resources["AWS::ElasticLoadBalancingV2::LoadBalancer"]
{
    "LoadBalancer": {
        "spec": {
            "name": "XXXXXXX",
            "type": "network",
            "scheme": "internet-facing",
            "ipAddressType": "ipv4",
            "subnetMapping": [
                {
                    "subnetID": "subnet-aaaaaaaaa"
                },
                {
                    "subnetID": "subnet-bbbbbbb"
                },
                {
                    "subnetID": "subnet-ggggggg"
                }
            ],
            "securityGroups": [
                {
                    "$ref": "#/resources/AWS::EC2::SecurityGroup/ManagedLBSecurityGroup/status/groupID"
                },
                "sg-TTTTTTTTTTTT"
            ]
        }
    }
}

Can you on given service.beta.kubernetes.io/aws-load-balancer-security-groups: '' not add this SGs to the model ? Maybe that will not be causing an error:

A load balancer with the same name 'XXXXXXX' exists, but with different settings.
oliviassss commented 4 months ago

@wbar, by 1.5.5 I suppose you mean helm chart version. From controller version v2.6.0, we support SGs for NLB and the controller will create front-end and back-end SGs and attach to the NLBs by default. If you want to opt-out, you can specify the feature gate flag --feature-gates=NLBSecurityGroup=false, or --set controllerConfig.featureGates.NLBSecurityGroup=false in helm cmd. see https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.7/deploy/configurations/#feature-gates

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 weeks ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten