no identity-based policy allows the elasticloadbalancing:AddTags

[dumlutimuralp]

Description

The current IAM policy published in the AWS Doc and AWS Load Balancer Controller GitHub page has incompatible definitions in it. This causes AWS Load Balancer Controller failing in provisioining an ELB due to some Addtag errors. Seems identical to this previous GitHub issue.

Versions

• Module Version [Required]:

• AWS Load Balancer controller 2.4.4 (also tested with 2.4.4)

• EKS v1.24

Reproduction Code [Required]

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: frontendnlb
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external #AWS Load Balancer Controller fulfills services that has this annotation 
    service.beta.kubernetes.io/aws-load-balancer-name : mynlb #User defined name given to AWS Network Load Balancer
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing #Places the load balancer on public subnets
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip #The Pod IPs should be used as the target IPs (rather than the node IPs as was the case with Network Load Balancer in the previous section)
  labels:
    app: frontendnlb
spec:
  type: LoadBalancer #The type of service
  ports:
    - port: 80 #The port which the service is running on
      targetPort: 9000 #The port on the pod which is backing this service. If not specified, it is assumed to be the same as the service port.
      name: http
  selector:
    app: frontend #The service identifies all the pods which have this label and then automatically configures the IP addresses of those pods as endpoints of this service.
EOF

Expected behaviour

For AWS Load Balancer Controller to provision an NLB.

Actual behaviour

No NLBs are provisioned. And AWS Load Balancer Controller generates the following error.

{"level":"error","ts":1676040723.3378003,"logger":"controller.service","msg":"Reconciler error","name":"frontendnlb","namespace":"workshop","error":"AccessDenied: User: arn:aws:sts::xxxxxxxxxxxx:assumed-role/eksctl-mycluster-addon-iamserviceaccount-kub-Role1-ALZQNS3HQWFQ/1676040699752964147 is not authorized to perform: elasticloadbalancing:AddTags on resource: arn:aws:elasticloadbalancing:eu-west-2:578334270511:targetgroup/k8s-workshop-frontend-8a6a890206/* because no identity-based policy allows the elasticloadbalancing:AddTags action\n\tstatus code: 403, request id: 4ee09784-239c-4afb-94f1-958b50134d8c"}

[Apollorion]

Strangely Im also getting this error. Its working in other environments but not in a new one I just spun up this morning.

[dumlutimuralp]

It seems like I submitted this issue in the wrong repo. Sorry. Will submit a new one in AWS Load Balancer Controller Repo.

[dumlutimuralp]

Submitted to the correct repo : https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/3044 Hence closing this one.

[armujahid]

Did you find any workaround of this issue?

I am facing this issue with aws-load-balancer-controller helm chart 1.4.8 v2.4.7 provisioned using v4.32.1 of https://github.com/aws-ia/terraform-aws-eks-blueprints

Ingress nginx failed to provision NLB because of this issue. (I had to reinstall ingress nginx to after upgrading eks from 1.23 to 1.27 and now I am stuck with this)

https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/3044#issuecomment-1607455313

[armujahid]

I had to temporarily remove this conditional block from clustername-lb-irsa IAM policy as a workaround to provision ingress nginx NLB

 {
            "Action": [
                "elasticloadbalancing:RemoveTags",
                "elasticloadbalancing:AddTags"
            ],
           - "Condition": {
            -    "Null": {
            -        "aws:RequestTag/elbv2.k8s.aws/cluster": "true",
            -        "aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
            -    }
           - },
            "Effect": "Allow",
            "Resource": [
                "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
                "arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
            ]
        },

Reference: https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/2692#issuecomment-1424991358

Edit: As per this comment https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/2692#issuecomment-1602615427 IAM policy has been updated. We need to ensure that we are using the latest policy in blueprints. This issue can be reopened and moved to terraform-aws-eks-blueprints-addons repo.

[Kampe]

This is still very much an issue and should be reopened, I used the same workaround to fix this.

[armujahid]

https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/issues/200 can be used to track this issue. Not sure weather the fix will be back ported to the v4 blueprints or not.

[m00lecule]

Today also faced this issue

[RyanDevlin]

I am also facing this issue when installing the Load Balancer Controller AddOn via EKS blueprints today.

[askulkarni2]

Tracking this in https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/issues/200

[MichaelHindley]

also ran into this with same versions as mentioned today

[SachithKasthuriarachchi]

ran into the same issue

[dcyoung]

+1 - battling this issue w/ deployment details:

eks version 1.27
source = "github.com/aws-ia/terraform-aws-eks-blueprints//modules/kubernetes-addons?ref=v4.32.1"
enable_aws_load_balancer_controller  = true
aws_load_balancer_controller_helm_config = {
    version                    = "1.5.5"
}

[michealliang123]

ran into the same issue today, I have two env,one is using LBC 2.4.7,another one using 2.6.0. They are using same policy,but version 2.6.0 has this error { "Effect": "Allow", "Action": [ "elasticloadbalancing:AddTags", "elasticloadbalancing:RemoveTags" ], "Resource": [ "arn:aws-cn:elasticloadbalancing:::targetgroup//", "arn:aws-cn:elasticloadbalancing:::loadbalancer/net//", "arn:aws-cn:elasticloadbalancing:::loadbalancer/app//" ], "Condition": { "Null": { "aws:RequestTag/elbv2.k8s.aws/cluster": "true", "aws:ResourceTag/elbv2.k8s.aws/cluster": "false" } } }, { "Effect": "Allow", "Action": [ "elasticloadbalancing:AddTags", "elasticloadbalancing:RemoveTags" ], "Resource": [ "arn:aws-cn:elasticloadbalancing:::listener/net///", "arn:aws-cn:elasticloadbalancing:::listener/app///", "arn:aws-cn:elasticloadbalancing:::listener-rule/net///", "arn:aws-cn:elasticloadbalancing:::listener-rule/app///"

[bryantbiggs]

in https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/pull/229 we have re-aligned the IAM policy with the policy specified in https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/docs/install/iam_policy.json

[omidraha]

FailedDeployModel  
ingress  Failed deploy model due to AccessDenied: 
User: arn:aws:sts::***:assumed-role/aws-loadbalancer-controller-role is not authorized to perform: 
elasticloadbalancing:AddTags on resource: arn:aws:elasticloadbalancing:us-west-2:***:targetgroup/k8s-appsdev-hybridtr-***/* 
because no identity-based policy allows the elasticloadbalancing:AddTags action

iam_policy.json: https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.4.0/docs/install/iam_policy.json

[Cyberphysics]

eks version v1.27.4-eks-2d98532 I degrade LBC from v1.6.0 to v1.5.3 by helm, but error still existed, and I use the workaround meshod which -+ condition in role policy, it dont work; then I upgrade the LBC to v1.5.4 by helm, it works! And I check the policy and found the removed statements come back, I dont know why, but it is resolved now.