search

aws / eks-charts

Amazon EKS Helm chart repository
Apache License 2.0
1.21k stars 976 forks source link

[AWS Load Balancer Controller ] failed to retrieve credentials caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity #507

Closed kaykhancheckpoint closed 3 years ago

kaykhancheckpoint commented 3 years ago

I am using the helm chart to install the aws load balancer controller.

https://github.com/aws/eks-charts/tree/master/stable/aws-load-balancer-controller

However when i apply the ingress controller i get the following error:

It looks like it is missing a permission, but the role i have created has the correct policy attached https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.1.2/docs/install/iam_policy.json

Can you check below if i am creating the correct role? as i was unsure about this bit

kubectl describe ing -n echoserver echoserver
Name:             echoserver
Namespace:        echoserver
Address:          
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host            Path  Backends
  ----            ----  --------
  echo.<redacted>.com  
                  /   echoserver:80 (10.0.1.188:8080)
Annotations:      alb.ingress.kubernetes.io/scheme: internet-facing
                  alb.ingress.kubernetes.io/tags: Environment=dev,Team=test
                  kubernetes.io/ingress.class: alb
Events:
  Type     Reason            Age   From     Message
  ----     ------            ----  ----     -------
  Warning  FailedBuildModel  39s   ingress  Failed build model due to couldn't auto-discover subnets: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
           status code: 403, request id: 40c9e27b-af7b-4e19-9ced-7fa46cbb7526
  Warning  FailedBuildModel  39s  ingress  Failed build model due to couldn't auto-discover subnets: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
           status code: 403, request id: 3003e79e-2585-4b36-9ac9-b2a1a8e961ce
  Warning  FailedBuildModel  39s  ingress  Failed build model due to couldn't auto-discover subnets: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
           status code: 403, request id: 6cae72f1-2a38-47a0-a485-685b9abfe451
  Warning  FailedBuildModel  38s  ingress  Failed build model due to couldn't auto-discover subnets: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
           status code: 403, request id: 7138d1ae-5dba-4604-8f5c-66ad2f2e5ba2
  Warning  FailedBuildModel  38s  ingress  Failed build model due to couldn't auto-discover subnets: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
           status code: 403, request id: 4f08528f-0f75-4e9b-a4eb-4148703d4560
  Warning  FailedBuildModel  38s  ingress  Failed build model due to couldn't auto-discover subnets: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
           status code: 403, request id: 09622f8c-d9ae-485a-a6fd-af6111a23d7c
  Warning  FailedBuildModel  37s  ingress  Failed build model due to couldn't auto-discover subnets: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
           status code: 403, request id: e42cbe7c-209f-429a-bbaa-c6e056eae69d
  Warning  FailedBuildModel  36s  ingress  Failed build model due to couldn't auto-discover subnets: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
           status code: 403, request id: 65ee4a19-6d94-4dc6-b90a-b73330cd579d
  Warning  FailedBuildModel  36s  ingress  Failed build model due to couldn't auto-discover subnets: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
           status code: 403, request id: 3a2c6830-db8d-42fd-b347-8b7caef77964
  Warning  FailedBuildModel  16s (x4 over 34s)  ingress  (combined from similar events): Failed build model due to couldn't auto-discover subnets: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
  status code: 403, request id: d3477675-9e44-4104-9539-63b8e017fc56

values.yml

# Default values for aws-load-balancer-controller.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

image:
  repository: 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-load-balancer-controller
  tag: v2.1.3
  pullPolicy: IfNotPresent

imagePullSecrets: []
nameOverride: "kube-system"
fullnameOverride: ""

# The name of the Kubernetes cluster. A non-empty value is required
clusterName: "<redacted>-prod-k8s"

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Annotations to add to the service account
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<redacted>:role/AWSLoadBalancerControllerIAMRole
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: "aws-load-balancer-controller"

rbac:
  # Specifies whether rbac resources should be created
  create: true

podSecurityContext:
  fsGroup: 65534

securityContext:
  # capabilities:
  #   drop:
  #   - ALL
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  allowPrivilegeEscalation: false

# Time period for the controller pod to do a graceful shutdown
terminationGracePeriodSeconds: 10

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  # requests:
  #   cpu: 100m
  #   memory: 128Mi

# Leverage a PriorityClass to ensure the controller will survive resource shortages
# ref: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/#priorityclass
priorityClassName: ""

nodeSelector: {}

tolerations: []

affinity: {}

podAnnotations: {}

podLabels: {}

# Enable cert-manager
enableCertManager: false

# The ingress class this controller will satisfy. If not specified, controller will match all
# ingresses without ingress class annotation and ingresses of type alb
ingressClass: alb

# The AWS region for the kubernetes cluster. Set to use KIAM or kube2iam for example.
region:

# The VPC ID for the Kubernetes cluster. Set this manually when your pods are unable to use the metadata service to determine this automatically
vpcId:

# Maximum retries for AWS APIs (default 10)
awsMaxRetries:

# If enabled, targetHealth readiness gate will get injected to the pod spec for the matching endpoint pods (default true)
enablePodReadinessGateInject:

# Enable Shield addon for ALB (default true)
enableShield:

# Enable WAF addon for ALB (default true)
enableWaf:

# Enable WAF V2 addon for ALB (default true)
enableWafv2:

# Maximum number of concurrently running reconcile loops for ingress (default 3)
ingressMaxConcurrentReconciles:

# Set the controller log level - info(default), debug (default "info")
logLevel:

# The address the metric endpoint binds to. (default ":8080")
metricsBindAddr: ""

# The TCP port the Webhook server binds to. (default 9443)
webhookBindPort:

# Maximum number of concurrently running reconcile loops for service (default 3)
serviceMaxConcurrentReconciles:

# Maximum number of concurrently running reconcile loops for targetGroupBinding
targetgroupbindingMaxConcurrentReconciles:

# Period at which the controller forces the repopulation of its local object stores. (default 1h0m0s)
syncPeriod:

# Namespace the controller watches for updates to Kubernetes objects, If empty, all namespaces are watched.
watchNamespace:

# Liveness probe configuration for the controller
livenessProbe:
  failureThreshold: 2
  httpGet:
    path: /healthz
    port: 61779
    scheme: HTTP
  initialDelaySeconds: 30
  timeoutSeconds: 10

# Environment variables to set for aws-load-balancer-controller pod.
# We strongly discourage programming access credentials in the controller environment. You should setup IRSA or
# comparable solutions like kube2iam, kiam etc instead.
env:
  # ENV_1: ""
  # ENV_2: ""

# Specifies if aws-load-balancer-controller should be started in hostNetwork mode.
#
# This is required if using a custom CNI where the managed control plane nodes are unable to initiate
# network connections to the pods, for example using Calico CNI plugin on EKS. This is not required or
# recommended if using the Amazon VPC CNI plugin.
hostNetwork: false

# extraVolumeMounts are the additional volume mounts. This enables setting up IRSA on non-EKS Kubernetes cluster
extraVolumeMounts:
  # - name: aws-iam-token
  #   mountPath: /var/run/secrets/eks.amazonaws.com/serviceaccount
  #   readOnly: true

# extraVolumes for the extraVolumeMounts. Useful to mount a projected service account token for example.
extraVolumes:
  # - name: aws-iam-token
  #   projected:
  #     defaultMode: 420
  #     sources:
  #     - serviceAccountToken:
  #         audience: sts.amazonaws.com
  #         expirationSeconds: 86400
  #         path: token

# defaultTags are the tags to apply to all AWS resources managed by this controller
defaultTags: {}
  # default_tag1: value1
  # default_tag2: value2

podDisruptionBudget: {}
#  maxUnavailable: 1

role creation

resource "aws_iam_policy" "AWSLoadBalancerControllerIAMPolicy" {
  name        = "AWSLoadBalancerControllerIAMPolicy"
  path        = "/"
  description = "AWS Load Balancer Controller Policy"

  # Terraform's "jsonencode" function converts a
  # Terraform expression result to valid JSON syntax.
  policy = file("k8sutils/alb-controller/iam-policy.json")

  tags = {
    Terraform   = "true"
    Environment = local.workspace
  }

}

resource "aws_iam_role" "AWSLoadBalancerControllerIAMRole" {
  name = "AWSLoadBalancerControllerIAMRole"

  tags = {
    Terraform   = "true"
    Environment = local.workspace
  }

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "AWSLoadBalancerControllerRolePolicAttachment" {
  role       = aws_iam_role.AWSLoadBalancerControllerIAMRole.name
  policy_arn = aws_iam_policy.AWSLoadBalancerControllerIAMPolicy.arn
}
kishorj commented 3 years ago

Duplicate of controller issue https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/1935. Since there is a fix mentioned in the closing comment, this no longer needs further investigation.