search

aws-ia / terraform-aws-eks-blueprints-addons

Terraform module which provisions addons on Amazon EKS clusters
https://aws-ia.github.io/terraform-aws-eks-blueprints-addons/main/
Apache License 2.0
247 stars 119 forks source link

AWS cluster autoscaler crashing #234

Closed jeenadeepak closed 1 year ago

jeenadeepak commented 1 year ago

Description

I have recently migrate terraform code from 4 to 5, But I am facing below issues with cluster autoscaling,

1: Cluster autoscaler is crashing, attached source code for reference 2: Cluster autoscaler pod name is "cluster-autoscaler-aws-cluster-autoscaler-67f5f76b5c-m5w5v" which contains "cluster-autoscaler" two times,

If your request is for a new feature, please use the Feature request template.

⚠️ Note

Before you submit an issue, please perform the following first:

  1. Remove the local .terraform directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): rm -rf .terraform/
  2. Re-initialize the project root to pull down modules: terraform init
  3. Re-attempt your terraform plan or apply and check if the issue still persists

Versions

Reproduction Code [Required]

module "eks_blueprints_addons" {
source = "aws-ia/eks-blueprints-addons/aws"
version = "~> 1.7.0" 

cluster_name      = module.eks.cluster_name
cluster_endpoint  = module.eks.cluster_endpoint
cluster_version   = module.eks.cluster_version
oidc_provider_arn = module.eks.oidc_provider

eks_addons = {
    aws-ebs-csi-driver = {
        most_recent = true
    }
    coredns = {
        most_recent = true
    }
    vpc-cni = {
        most_recent = true
    }
    kube-proxy = {
        most_recent = true
    }
}

enable_aws_load_balancer_controller    = var.enable_eks_aws_load_balancer_controller
enable_metrics_server                  = var.enable_eks_metrics_server
enable_cluster_autoscaler              = var.enable_eks_cluster_autoscaler

enable_cert_manager                    = var.enable_eks_cert_manager
cert_manager_route53_hosted_zone_arns  = ["arn:aws:route53:::hostedzone/${local.route53_hosted_zone_id}"]

}

Steps to reproduce the behavior:

terraform init terraform apply

Expected behaviour

Actual behaviour

Terminal Output Screenshot(s)

Screenshot 2023-08-20 at 3 33 28 AM

Logs:

kubectl logs cluster-autoscaler-aws-cluster-autoscaler-67f5f76b5c-m5w5v -n kube-system -f I0819 22:00:55.774751 1 flags.go:57] FLAG: --add-dir-header="false" I0819 22:00:55.774781 1 flags.go:57] FLAG: --address=":8085" I0819 22:00:55.774785 1 flags.go:57] FLAG: --alsologtostderr="false" I0819 22:00:55.774789 1 flags.go:57] FLAG: --aws-use-static-instance-list="false" I0819 22:00:55.774792 1 flags.go:57] FLAG: --balance-similar-node-groups="false" I0819 22:00:55.774795 1 flags.go:57] FLAG: --balancing-ignore-label="[]" I0819 22:00:55.774801 1 flags.go:57] FLAG: --balancing-label="[]" I0819 22:00:55.774803 1 flags.go:57] FLAG: --cloud-config="" I0819 22:00:55.774806 1 flags.go:57] FLAG: --cloud-provider="aws" I0819 22:00:55.774810 1 flags.go:57] FLAG: --cloud-provider-gce-l7lb-src-cidrs="130.211.0.0/22,35.191.0.0/16" I0819 22:00:55.774816 1 flags.go:57] FLAG: --cloud-provider-gce-lb-src-cidrs="130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16" I0819 22:00:55.774822 1 flags.go:57] FLAG: --cluster-name="" I0819 22:00:55.774826 1 flags.go:57] FLAG: --clusterapi-cloud-config-authoritative="false" I0819 22:00:55.774830 1 flags.go:57] FLAG: --cordon-node-before-terminating="false" I0819 22:00:55.774834 1 flags.go:57] FLAG: --cores-total="0:320000" I0819 22:00:55.774839 1 flags.go:57] FLAG: --daemonset-eviction-for-empty-nodes="false" I0819 22:00:55.774843 1 flags.go:57] FLAG: --daemonset-eviction-for-occupied-nodes="true" I0819 22:00:55.774846 1 flags.go:57] FLAG: --debugging-snapshot-enabled="false" I0819 22:00:55.774849 1 flags.go:57] FLAG: --emit-per-nodegroup-metrics="false" I0819 22:00:55.774851 1 flags.go:57] FLAG: --enforce-node-group-min-size="false" I0819 22:00:55.774855 1 flags.go:57] FLAG: --estimator="binpacking" I0819 22:00:55.774858 1 flags.go:57] FLAG: --expander="random" I0819 22:00:55.774861 1 flags.go:57] FLAG: --expendable-pods-priority-cutoff="-10" I0819 22:00:55.774865 1 flags.go:57] FLAG: --feature-gates="" I0819 22:00:55.774873 1 flags.go:57] FLAG: --force-ds="false" I0819 22:00:55.774877 1 flags.go:57] FLAG: --gce-concurrent-refreshes="1" I0819 22:00:55.774882 1 flags.go:57] FLAG: --gce-expander-ephemeral-storage-support="false" I0819 22:00:55.774887 1 flags.go:57] FLAG: --gce-mig-instances-min-refresh-wait-time="5s" I0819 22:00:55.774890 1 flags.go:57] FLAG: --gpu-total="[]" I0819 22:00:55.774893 1 flags.go:57] FLAG: --grpc-expander-cert="" I0819 22:00:55.774896 1 flags.go:57] FLAG: --grpc-expander-url="" I0819 22:00:55.774900 1 flags.go:57] FLAG: --ignore-daemonsets-utilization="false" I0819 22:00:55.774904 1 flags.go:57] FLAG: --ignore-mirror-pods-utilization="false" I0819 22:00:55.774908 1 flags.go:57] FLAG: --ignore-taint="[]" I0819 22:00:55.774913 1 flags.go:57] FLAG: --initial-node-group-backoff-duration="5m0s" I0819 22:00:55.774918 1 flags.go:57] FLAG: --kube-client-burst="10" I0819 22:00:55.774922 1 flags.go:57] FLAG: --kube-client-qps="5" I0819 22:00:55.774930 1 flags.go:57] FLAG: --kubeconfig="" I0819 22:00:55.774936 1 flags.go:57] FLAG: --kubernetes="" I0819 22:00:55.774941 1 flags.go:57] FLAG: --leader-elect="true" I0819 22:00:55.774958 1 flags.go:57] FLAG: --leader-elect-lease-duration="15s" I0819 22:00:55.774962 1 flags.go:57] FLAG: --leader-elect-renew-deadline="10s" I0819 22:00:55.774965 1 flags.go:57] FLAG: --leader-elect-resource-lock="leases" I0819 22:00:55.774969 1 flags.go:57] FLAG: --leader-elect-resource-name="cluster-autoscaler" I0819 22:00:55.774972 1 flags.go:57] FLAG: --leader-elect-resource-namespace="" I0819 22:00:55.774975 1 flags.go:57] FLAG: --leader-elect-retry-period="2s" I0819 22:00:55.774978 1 flags.go:57] FLAG: --log-backtrace-at=":0" I0819 22:00:55.774983 1 flags.go:57] FLAG: --log-dir="" I0819 22:00:55.774987 1 flags.go:57] FLAG: --log-file="" I0819 22:00:55.774990 1 flags.go:57] FLAG: --log-file-max-size="1800" I0819 22:00:55.774994 1 flags.go:57] FLAG: --logtostderr="true" I0819 22:00:55.774999 1 flags.go:57] FLAG: --max-allocatable-difference-ratio="0.05" I0819 22:00:55.775004 1 flags.go:57] FLAG: --max-autoprovisioned-node-group-count="15" I0819 22:00:55.775009 1 flags.go:57] FLAG: --max-bulk-soft-taint-count="10" I0819 22:00:55.775013 1 flags.go:57] FLAG: --max-bulk-soft-taint-time="3s" I0819 22:00:55.775019 1 flags.go:57] FLAG: --max-drain-parallelism="1" I0819 22:00:55.775023 1 flags.go:57] FLAG: --max-empty-bulk-delete="10" I0819 22:00:55.775027 1 flags.go:57] FLAG: --max-failing-time="15m0s" I0819 22:00:55.775032 1 flags.go:57] FLAG: --max-free-difference-ratio="0.05" I0819 22:00:55.775037 1 flags.go:57] FLAG: --max-graceful-termination-sec="600" I0819 22:00:55.775049 1 flags.go:57] FLAG: --max-inactivity="10m0s" I0819 22:00:55.775054 1 flags.go:57] FLAG: --max-node-group-backoff-duration="30m0s" I0819 22:00:55.775058 1 flags.go:57] FLAG: --max-node-provision-time="15m0s" I0819 22:00:55.775063 1 flags.go:57] FLAG: --max-nodegroup-binpacking-duration="10s" I0819 22:00:55.775067 1 flags.go:57] FLAG: --max-nodes-per-scaleup="1000" I0819 22:00:55.775072 1 flags.go:57] FLAG: --max-nodes-total="0" I0819 22:00:55.775077 1 flags.go:57] FLAG: --max-pod-eviction-time="2m0s" I0819 22:00:55.775082 1 flags.go:57] FLAG: --max-scale-down-parallelism="10" I0819 22:00:55.775086 1 flags.go:57] FLAG: --max-total-unready-percentage="45" I0819 22:00:55.775092 1 flags.go:57] FLAG: --memory-difference-ratio="0.015" I0819 22:00:55.775097 1 flags.go:57] FLAG: --memory-total="0:6400000" I0819 22:00:55.775102 1 flags.go:57] FLAG: --min-replica-count="0" I0819 22:00:55.775106 1 flags.go:57] FLAG: --namespace="kube-system" I0819 22:00:55.775109 1 flags.go:57] FLAG: --new-pod-scale-up-delay="0s" I0819 22:00:55.775113 1 flags.go:57] FLAG: --node-autoprovisioning-enabled="false" I0819 22:00:55.775117 1 flags.go:57] FLAG: --node-delete-delay-after-taint="5s" I0819 22:00:55.775121 1 flags.go:57] FLAG: --node-deletion-batcher-interval="0s" I0819 22:00:55.775124 1 flags.go:57] FLAG: --node-deletion-delay-timeout="2m0s" I0819 22:00:55.775127 1 flags.go:57] FLAG: --node-group-auto-discovery="[asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/dev-eks]" I0819 22:00:55.775133 1 flags.go:57] FLAG: --node-group-backoff-reset-timeout="3h0m0s" I0819 22:00:55.775136 1 flags.go:57] FLAG: --node-info-cache-expire-time="87600h0m0s" I0819 22:00:55.775140 1 flags.go:57] FLAG: --nodes="[]" I0819 22:00:55.775143 1 flags.go:57] FLAG: --ok-total-unready-count="3" I0819 22:00:55.775147 1 flags.go:57] FLAG: --one-output="false" I0819 22:00:55.775150 1 flags.go:57] FLAG: --parallel-drain="false" I0819 22:00:55.775154 1 flags.go:57] FLAG: --profiling="false" I0819 22:00:55.775159 1 flags.go:57] FLAG: --record-duplicated-events="false" I0819 22:00:55.775163 1 flags.go:57] FLAG: --regional="false" I0819 22:00:55.775168 1 flags.go:57] FLAG: --scale-down-candidates-pool-min-count="50" I0819 22:00:55.775172 1 flags.go:57] FLAG: --scale-down-candidates-pool-ratio="0.1" I0819 22:00:55.775177 1 flags.go:57] FLAG: --scale-down-delay-after-add="10m0s" I0819 22:00:55.775183 1 flags.go:57] FLAG: --scale-down-delay-after-delete="0s" I0819 22:00:55.775188 1 flags.go:57] FLAG: --scale-down-delay-after-failure="3m0s" I0819 22:00:55.775194 1 flags.go:57] FLAG: --scale-down-enabled="true" I0819 22:00:55.775198 1 flags.go:57] FLAG: --scale-down-gpu-utilization-threshold="0.5" I0819 22:00:55.775203 1 flags.go:57] FLAG: --scale-down-non-empty-candidates-count="30" I0819 22:00:55.775207 1 flags.go:57] FLAG: --scale-down-simulation-timeout="30s" I0819 22:00:55.775212 1 flags.go:57] FLAG: --scale-down-unneeded-time="10m0s" I0819 22:00:55.775215 1 flags.go:57] FLAG: --scale-down-unready-enabled="true" I0819 22:00:55.775218 1 flags.go:57] FLAG: --scale-down-unready-time="20m0s" I0819 22:00:55.775221 1 flags.go:57] FLAG: --scale-down-utilization-threshold="0.5" I0819 22:00:55.775224 1 flags.go:57] FLAG: --scale-up-from-zero="true" I0819 22:00:55.775228 1 flags.go:57] FLAG: --scan-interval="10s" I0819 22:00:55.775232 1 flags.go:57] FLAG: --skip-headers="false" I0819 22:00:55.775236 1 flags.go:57] FLAG: --skip-log-headers="false" I0819 22:00:55.775241 1 flags.go:57] FLAG: --skip-nodes-with-custom-controller-pods="true" I0819 22:00:55.775245 1 flags.go:57] FLAG: --skip-nodes-with-local-storage="true" I0819 22:00:55.775249 1 flags.go:57] FLAG: --skip-nodes-with-system-pods="true" I0819 22:00:55.775254 1 flags.go:57] FLAG: --status-config-map-name="cluster-autoscaler-status" I0819 22:00:55.775257 1 flags.go:57] FLAG: --stderrthreshold="0" I0819 22:00:55.775260 1 flags.go:57] FLAG: --unremovable-node-recheck-timeout="5m0s" I0819 22:00:55.775263 1 flags.go:57] FLAG: --user-agent="cluster-autoscaler" I0819 22:00:55.775266 1 flags.go:57] FLAG: --v="4" I0819 22:00:55.775268 1 flags.go:57] FLAG: --vmodule="" I0819 22:00:55.775271 1 flags.go:57] FLAG: --write-status-configmap="true" I0819 22:00:55.775279 1 main.go:534] Cluster Autoscaler 1.27.3 I0819 22:00:55.795933 1 leaderelection.go:245] attempting to acquire leader lease kube-system/cluster-autoscaler... I0819 22:00:55.810897 1 leaderelection.go:255] successfully acquired lease kube-system/cluster-autoscaler I0819 22:00:55.811044 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"Lease", Namespace:"kube-system", Name:"cluster-autoscaler", UID:"ac9d54ca-6e39-4915-9a3e-cdb54675e1a2", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"136821", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' cluster-autoscaler-aws-cluster-autoscaler-67f5f76b5c-m5w5v became leader I0819 22:00:55.812750 1 reflector.go:287] Starting reflector v1.CSINode (0s) from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.812767 1 reflector.go:323] Listing and watching v1.CSINode from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.812771 1 reflector.go:287] Starting reflector v1.CSIDriver (0s) from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.812783 1 reflector.go:323] Listing and watching v1.CSIDriver from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.812789 1 reflector.go:287] Starting reflector v1.PersistentVolume (0s) from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.812797 1 reflector.go:323] Listing and watching v1.PersistentVolume from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.812954 1 reflector.go:287] Starting reflector v1.PersistentVolumeClaim (0s) from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.812962 1 reflector.go:323] Listing and watching v1.PersistentVolumeClaim from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.812963 1 reflector.go:287] Starting reflector v1.ReplicationController (0s) from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.812976 1 reflector.go:323] Listing and watching v1.ReplicationController from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.812970 1 reflector.go:287] Starting reflector v1.StatefulSet (0s) from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.812989 1 reflector.go:323] Listing and watching v1.StatefulSet from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.813053 1 reflector.go:287] Starting reflector v1.Pod (0s) from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.813060 1 reflector.go:323] Listing and watching v1.Pod from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.813137 1 reflector.go:287] Starting reflector v1.Node (0s) from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.813143 1 reflector.go:323] Listing and watching v1.Node from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.813233 1 reflector.go:287] Starting reflector v1.PodDisruptionBudget (0s) from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.813246 1 reflector.go:323] Listing and watching v1.PodDisruptionBudget from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.813360 1 reflector.go:287] Starting reflector v1.ReplicaSet (0s) from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.813369 1 reflector.go:323] Listing and watching v1.ReplicaSet from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.813401 1 reflector.go:287] Starting reflector v1.CSIStorageCapacity (0s) from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.813409 1 reflector.go:323] Listing and watching v1.CSIStorageCapacity from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.813447 1 reflector.go:287] Starting reflector v1.StorageClass (0s) from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.813450 1 reflector.go:287] Starting reflector v1.Service (0s) from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.813455 1 reflector.go:323] Listing and watching v1.StorageClass from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.813457 1 reflector.go:323] Listing and watching v1.Service from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.813360 1 reflector.go:287] Starting reflector v1.Namespace (0s) from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.813532 1 reflector.go:323] Listing and watching v1.Namespace from k8s.io/client-go/informers/factory.go:150 I0819 22:00:55.813536 1 reflector.go:287] Starting reflector v1.Pod (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:188 I0819 22:00:55.813543 1 reflector.go:323] Listing and watching v1.Pod from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:188 I0819 22:00:55.813524 1 reflector.go:287] Starting reflector v1.DaemonSet (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:320 I0819 22:00:55.813570 1 reflector.go:323] Listing and watching v1.DaemonSet from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:320 I0819 22:00:55.813608 1 reflector.go:287] Starting reflector v1.Pod (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:212 I0819 22:00:55.813614 1 reflector.go:323] Listing and watching v1.Pod from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:212 I0819 22:00:55.813615 1 reflector.go:287] Starting reflector v1.ReplicationController (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:329 I0819 22:00:55.813623 1 reflector.go:323] Listing and watching v1.ReplicationController from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:329 I0819 22:00:55.813624 1 reflector.go:287] Starting reflector v1.Job (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:338 I0819 22:00:55.813628 1 reflector.go:287] Starting reflector v1.Node (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:246 I0819 22:00:55.813631 1 reflector.go:323] Listing and watching v1.Job from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:338 I0819 22:00:55.813642 1 reflector.go:323] Listing and watching v1.Node from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:246 I0819 22:00:55.813710 1 reflector.go:287] Starting reflector v1.Node (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:246 I0819 22:00:55.813712 1 reflector.go:287] Starting reflector v1.StatefulSet (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:356 I0819 22:00:55.813719 1 reflector.go:323] Listing and watching v1.StatefulSet from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:356 I0819 22:00:55.813614 1 reflector.go:287] Starting reflector v1.ReplicaSet (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:347 I0819 22:00:55.813724 1 reflector.go:287] Starting reflector v1.PodDisruptionBudget (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309 I0819 22:00:55.813729 1 reflector.go:323] Listing and watching v1.ReplicaSet from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:347 I0819 22:00:55.813731 1 reflector.go:323] Listing and watching v1.PodDisruptionBudget from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309 I0819 22:00:55.813722 1 reflector.go:323] Listing and watching v1.Node from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:246 I0819 22:00:56.013320 1 request.go:628] Waited for 199.807932ms due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/storage.k8s.io/v1/csistoragecapacities?limit=500&resourceVersion=0 I0819 22:00:56.213184 1 request.go:628] Waited for 399.6729ms due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apps/v1/replicasets?limit=500&resourceVersion=0 I0819 22:00:56.413024 1 request.go:628] Waited for 599.484128ms due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0 I0819 22:00:56.613293 1 request.go:628] Waited for 799.726651ms due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/api/v1/services?limit=500&resourceVersion=0 I0819 22:00:56.813095 1 request.go:628] Waited for 999.45065ms due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0 I0819 22:00:57.013149 1 request.go:628] Waited for 1.199513742s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apps/v1/daemonsets?limit=500&resourceVersion=0 I0819 22:00:57.013194 1 request.go:696] Waited for 1.199513742s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apps/v1/daemonsets?limit=500&resourceVersion=0 I0819 22:00:57.213576 1 request.go:628] Waited for 1.399925195s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0 I0819 22:00:57.413452 1 request.go:628] Waited for 1.599707283s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/api/v1/nodes?limit=500&resourceVersion=0 I0819 22:00:57.613724 1 request.go:628] Waited for 1.799986201s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/api/v1/replicationcontrollers?limit=500&resourceVersion=0 I0819 22:00:57.813279 1 request.go:628] Waited for 1.99952278s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&limit=500&resourceVersion=0 I0819 22:00:58.013076 1 request.go:628] Waited for 2.199298118s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/batch/v1/jobs?limit=500&resourceVersion=0 I0819 22:00:58.213625 1 request.go:628] Waited for 2.399781722s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apps/v1/replicasets?limit=500&resourceVersion=0 I0819 22:00:58.213650 1 request.go:696] Waited for 2.399781722s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apps/v1/replicasets?limit=500&resourceVersion=0 I0819 22:00:58.413022 1 request.go:628] Waited for 2.599208375s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/policy/v1/poddisruptionbudgets?limit=500&resourceVersion=0 I0819 22:00:58.613817 1 request.go:628] Waited for 2.799978447s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/api/v1/nodes?limit=500&resourceVersion=0 I0819 22:00:58.813692 1 request.go:628] Waited for 2.999869096s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0 I0819 22:00:59.013481 1 request.go:628] Waited for 3.191131595s due to client-side throttling, not priority and fairness, request: PUT:https://172.20.0.1:443/api/v1/namespaces/kube-system/configmaps/cluster-autoscaler-status I0819 22:00:59.024321 1 cloud_provider_builder.go:29] Building aws cloud provider. E0819 22:00:59.064690 1 aws_cloud_provider.go:410] Failed to generate AWS EC2 Instance Types: WebIdentityErr: failed to retrieve credentials caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity status code: 403, request id: 7383e57d-6da5-4d1f-a28f-0125310b9322, falling back to static list with last update time: 2023-02-14 I0819 22:00:59.069916 1 aws_cloud_provider.go:432] Successfully load 630 EC2 Instance Types [c6gn.large r5n.4xlarge r6a.24xlarge d3.8xlarge t3a.2xlarge c7g.metal r6g.medium m6gd.16xlarge h1.8xlarge r5b.xlarge m6i.metal r6a.8xlarge is4gen.8xlarge c4.2xlarge u-3tb1.56xlarge m6a.xlarge m7g.4xlarge a1.medium m5ad.24xlarge m1.medium x2iedn.32xlarge m6i.8xlarge m6a.large r5.metal c5ad.xlarge m7g.xlarge r6i.8xlarge r4.xlarge c6g.16xlarge r5n.12xlarge r5.12xlarge c5ad.4xlarge c6gd.large x2iedn.2xlarge mac1.metal r6idn.12xlarge d2.2xlarge c6id.16xlarge c6in.xlarge x2gd.8xlarge m6g.medium r6a.32xlarge r7g.large c5a.16xlarge m6in.8xlarge i4i.large m6id.large r5ad.large m6in.2xlarge x1e.8xlarge m6idn.12xlarge m5n.24xlarge m5dn.12xlarge m5n.metal g5g.8xlarge c6i.xlarge i3en.xlarge p3dn.24xlarge g4dn.metal r6i.4xlarge m3.2xlarge t2.large c3.4xlarge c7g.16xlarge r5n.xlarge r6id.2xlarge c6in.32xlarge r5ad.24xlarge c5d.large r6i.12xlarge r6idn.large d2.xlarge g3.4xlarge t2.medium m6a.32xlarge u-6tb1.112xlarge m3.medium r6a.xlarge g3s.xlarge r6idn.4xlarge r6in.8xlarge r5b.4xlarge m5ad.12xlarge dl1.24xlarge d2.8xlarge c3.8xlarge m6i.16xlarge r6g.xlarge i3.metal c6a.8xlarge c6g.2xlarge r6gd.metal m6g.16xlarge r5b.large r6in.12xlarge r6id.xlarge c4.xlarge m5.metal i4i.xlarge p2.8xlarge m6i.xlarge z1d.6xlarge x2iezn.2xlarge m6in.4xlarge t3a.nano c6a.xlarge x1.32xlarge m5dn.16xlarge r5n.8xlarge c5n.4xlarge m5dn.8xlarge t4g.2xlarge x2iezn.12xlarge c6gd.16xlarge t3.medium x2iedn.xlarge m6gd.xlarge u-9tb1.112xlarge r5d.large c5d.18xlarge r5ad.4xlarge m5d.16xlarge c6a.16xlarge r6i.32xlarge r5.xlarge g2.8xlarge r6gd.8xlarge m6in.large r5a.12xlarge x2iedn.16xlarge m6g.12xlarge x2gd.large inf1.24xlarge c6id.4xlarge x2iedn.metal m5a.xlarge m6a.24xlarge d3en.xlarge c5d.metal m6g.4xlarge m7g.metal d3.xlarge x2gd.xlarge m7g.8xlarge a1.metal c6id.12xlarge i3.4xlarge r5d.24xlarge r5dn.metal i4i.32xlarge p3.8xlarge r6g.large r6in.4xlarge c6gn.xlarge t3a.micro r6g.12xlarge is4gen.medium inf1.xlarge i3en.24xlarge m5dn.xlarge r6i.metal r5n.16xlarge r5d.2xlarge m6idn.2xlarge m6id.12xlarge r4.2xlarge c6a.24xlarge m1.large m7g.large m6id.2xlarge m5dn.24xlarge r4.16xlarge c6gn.16xlarge d3en.8xlarge f1.2xlarge c6a.32xlarge g5g.16xlarge u-24tb1.112xlarge t3.nano r5b.2xlarge im4gn.16xlarge g4dn.4xlarge c5d.12xlarge inf2.8xlarge c5.18xlarge r6i.16xlarge r5a.xlarge r5n.large c1.xlarge c6in.large m5ad.16xlarge z1d.12xlarge z1d.large m7g.2xlarge m5dn.metal c5.metal g5.16xlarge c7g.medium r6idn.24xlarge z1d.2xlarge c5d.9xlarge x2gd.2xlarge i4i.metal g5g.4xlarge r6idn.16xlarge c5n.large c6a.4xlarge r7g.4xlarge r5d.12xlarge g5.24xlarge m5.16xlarge i3.16xlarge x1e.32xlarge m6i.4xlarge c5n.2xlarge r5b.24xlarge r6id.16xlarge r7g.xlarge t4g.medium r5d.metal r5a.2xlarge m5.12xlarge r5.2xlarge g4ad.8xlarge r6g.2xlarge c5a.4xlarge c7g.8xlarge r5.8xlarge d3en.4xlarge r4.large m5a.2xlarge c3.large p2.xlarge i4i.8xlarge g2.2xlarge t3a.medium m3.large m6a.48xlarge r3.xlarge m5n.16xlarge r3.8xlarge r5d.xlarge c4.large m3.xlarge m5dn.2xlarge r6i.large d3.2xlarge r5d.16xlarge t3.xlarge g4dn.16xlarge r6gd.xlarge i3en.12xlarge m6gd.8xlarge m6g.large x2gd.metal m6idn.24xlarge m6idn.16xlarge c6i.24xlarge m5d.24xlarge i4i.4xlarge t2.nano g4ad.4xlarge c6g.medium z1d.3xlarge r6gd.large g3.16xlarge t4g.xlarge c5a.12xlarge m5ad.xlarge g4ad.2xlarge u-6tb1.56xlarge m5d.4xlarge r6a.metal m5n.12xlarge r6a.12xlarge i4i.2xlarge c6a.2xlarge c6g.4xlarge r6a.16xlarge c6gn.4xlarge r5d.8xlarge t4g.nano c6gn.8xlarge m1.xlarge r6g.4xlarge im4gn.8xlarge c6g.8xlarge m6id.16xlarge c5a.large m6i.2xlarge m5d.8xlarge c6gd.4xlarge c6i.4xlarge z1d.metal i2.xlarge c7g.12xlarge im4gn.4xlarge m5d.metal m5a.16xlarge x2idn.32xlarge c5n.metal r5ad.2xlarge m6id.24xlarge r6id.4xlarge g5.8xlarge r7g.12xlarge r5a.24xlarge x2idn.24xlarge g5.xlarge m5d.large c7g.2xlarge r5b.12xlarge m4.10xlarge r6in.large m5.24xlarge m5ad.4xlarge r5dn.4xlarge g4dn.8xlarge m6in.xlarge r6idn.8xlarge c5n.xlarge c6id.2xlarge r6in.32xlarge m5ad.8xlarge m6id.xlarge c6gd.xlarge r3.2xlarge c6id.metal r5b.metal r6idn.xlarge r6g.8xlarge m5n.4xlarge r5n.metal m6a.2xlarge c6i.32xlarge m7g.medium m6i.large cc2.8xlarge u-12tb1.112xlarge r5n.24xlarge m5.8xlarge c5n.18xlarge c6gd.12xlarge m5a.24xlarge c5a.24xlarge m5a.8xlarge r5ad.xlarge c6i.16xlarge c6id.8xlarge m6g.xlarge c1.medium r6in.2xlarge p4d.24xlarge m6gd.12xlarge is4gen.2xlarge r5dn.2xlarge t3.micro c6id.24xlarge c6gd.metal c6a.12xlarge m2.2xlarge c6id.32xlarge m6gd.medium x2iezn.4xlarge m6idn.4xlarge mac2.metal c6i.metal r6i.xlarge p3.16xlarge c6gn.medium t4g.micro x1e.16xlarge g4dn.2xlarge h1.2xlarge i4i.16xlarge r7g.2xlarge m5dn.large r5dn.12xlarge is4gen.4xlarge c6gn.2xlarge r5b.16xlarge h1.16xlarge r6i.24xlarge inf1.2xlarge c5ad.8xlarge c6gd.medium m5n.8xlarge p4de.24xlarge c6gn.12xlarge r5a.16xlarge m4.xlarge r5dn.large m6i.24xlarge c5ad.12xlarge is4gen.xlarge m6in.24xlarge m6in.12xlarge x2iedn.4xlarge c5a.8xlarge r6gd.16xlarge m6a.8xlarge is4gen.large i2.4xlarge t4g.large x1e.2xlarge c6in.16xlarge c5d.xlarge a1.4xlarge c6g.metal vt1.3xlarge c6g.xlarge r5b.8xlarge m4.16xlarge inf2.48xlarge m5n.2xlarge r6in.24xlarge t3.small r6gd.medium i3.large g5g.2xlarge m6g.2xlarge r7g.medium r5dn.24xlarge g4dn.12xlarge m5n.large c5ad.large m6idn.32xlarge m6g.metal m6idn.xlarge d3.4xlarge m6id.metal m6in.32xlarge c5ad.24xlarge c7g.xlarge m5ad.large m6idn.large g5g.xlarge m6g.8xlarge m6in.16xlarge m5.2xlarge inf2.xlarge p3.2xlarge trn1.2xlarge a1.large m6i.32xlarge c6in.12xlarge r6gd.4xlarge c5a.xlarge m4.large m5d.xlarge r7g.metal r6idn.2xlarge i3.8xlarge x2iezn.6xlarge t2.xlarge m1.small r5a.large a1.2xlarge c5.24xlarge t3a.large t2.2xlarge m5a.4xlarge c6in.2xlarge r6g.metal p2.16xlarge m5d.2xlarge c6i.12xlarge d3en.6xlarge m5zn.6xlarge c5.12xlarge i2.8xlarge g5.12xlarge m2.4xlarge c6a.48xlarge c5ad.16xlarge c5d.4xlarge g5g.metal c6i.large c6i.8xlarge m6id.32xlarge r6a.4xlarge x2gd.16xlarge c6a.large r5ad.16xlarge m5n.xlarge m5.xlarge inf1.6xlarge im4gn.large g4ad.16xlarge x2idn.metal c6gd.2xlarge c6id.xlarge x1e.4xlarge r5.24xlarge i3en.large m6i.12xlarge c6id.large i2.2xlarge m5zn.large t4g.small i3en.6xlarge c6in.8xlarge c5.2xlarge c6a.metal c4.4xlarge r6in.16xlarge r7g.16xlarge r6i.2xlarge r7g.8xlarge m6gd.4xlarge r6a.48xlarge c6g.large r6id.24xlarge t3.large r4.4xlarge c5a.2xlarge c6g.12xlarge x2gd.12xlarge r6a.large x2iezn.metal r6gd.2xlarge trn1.32xlarge r5.16xlarge c7g.large r6idn.32xlarge g5.48xlarge f1.4xlarge inf2.24xlarge t3a.xlarge m5zn.3xlarge r3.large c6in.24xlarge c5.9xlarge u-18tb1.112xlarge m5zn.xlarge c5.large r6id.12xlarge r3.4xlarge r5a.4xlarge im4gn.xlarge m6a.metal m6gd.large m6a.4xlarge c3.xlarge t2.micro r5.large c7g.4xlarge r5ad.8xlarge d3en.2xlarge c6gd.8xlarge im4gn.2xlarge r5d.4xlarge m5zn.12xlarge r6gd.12xlarge m4.4xlarge c4.8xlarge f1.16xlarge x2gd.medium i3.2xlarge g5.4xlarge m6gd.2xlarge c3.2xlarge t3a.small c6in.4xlarge g4ad.xlarge g3.8xlarge r5dn.8xlarge m5zn.metal c5.4xlarge c5d.24xlarge x1.16xlarge m5dn.4xlarge r6id.large r6a.2xlarge z1d.xlarge m5a.12xlarge m2.xlarge i3en.metal i3en.2xlarge a1.xlarge c6i.2xlarge d2.4xlarge t1.micro r6id.metal m4.2xlarge r5.4xlarge i3.xlarge m6gd.metal r4.8xlarge g4dn.xlarge c5.xlarge x2iedn.8xlarge x1e.xlarge i3en.3xlarge m6idn.8xlarge r5dn.16xlarge m5zn.2xlarge r5dn.xlarge vt1.6xlarge m6id.4xlarge m5ad.2xlarge m6a.16xlarge m5d.12xlarge d3en.12xlarge m7g.16xlarge x2gd.4xlarge t2.small r6id.8xlarge r5n.2xlarge m5.4xlarge m5a.large c5d.2xlarge h1.4xlarge r6g.16xlarge x2iezn.8xlarge r5ad.12xlarge t3.2xlarge vt1.24xlarge x2idn.16xlarge m7g.12xlarge r6in.xlarge x2iedn.24xlarge m6id.8xlarge m5.large c5ad.2xlarge r5a.8xlarge g5.2xlarge c5n.9xlarge m6a.12xlarge r6id.32xlarge] I0819 22:00:59.070123 1 auto_scaling_groups.go:367] Regenerating instance to ASG map for ASG names: [] I0819 22:00:59.070142 1 auto_scaling_groups.go:374] Regenerating instance to ASG map for ASG tags: map[k8s.io/cluster-autoscaler/dev-eks: k8s.io/cluster-autoscaler/enabled:] E0819 22:00:59.077057 1 aws_manager.go:125] Failed to regenerate ASG cache: WebIdentityErr: failed to retrieve credentials caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity status code: 403, request id: 503ec598-c704-4229-be93-78938b8143bb F0819 22:00:59.077082 1 aws_cloud_provider.go:437] Failed to create AWS Manager: WebIdentityErr: failed to retrieve credentials caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity status code: 403, request id: 503ec598-c704-4229-be93-78938b8143bb

Additional context

bryantbiggs commented 1 year ago

Can you provide a full reproduction that's deployable please?

jeenadeepak commented 1 year ago

@bryantbiggs : I can't share full deployable code as I have create terraform deployment code such that one is depends on other module,for example, I have a different module vpc which create a lot of resource as per my organisation need, and that module output is used for eks module, while migrating I have just replaced eks blueprint with new code as per document, and same for addons, , sharing eks module I am using to create eks cluster, can you replace local variable with respective code.

EKS CLUSTER


module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.16.0"

  cluster_name              = local.cluster_name
  cluster_version           = local.cluster_version
  vpc_id                    = local.vpc_id

  subnet_ids                = local.private_subnet_ids    # Enter Private Subnet IDs

  cluster_endpoint_public_access  = true
  cluster_endpoint_private_access = true

  manage_aws_auth_configmap = true

  aws_auth_roles = local.aws_auth_roles

  node_security_group_id = local.database_inbound_sg_id

  cluster_security_group_name = "${local.cluster_name}-cluster-sg"

  eks_managed_node_groups = {
    green = {

      name = "${local.cluster_name}-ng-green"
      use_name_prefix = true

      iam_role_name = "${local.cluster_name}-ng-green"

      launch_template_use_name_prefix = true
      launch_template_name = "${local.cluster_name}-ng-green"

      use_custom_launch_template = false

      capacity_type = "SPOT"
      instance_types  = ["t3.xlarge"]

      subnet_ids      = local.private_subnet_ids

      max_size = var.max_node_count
      min_size = var.min_node_count
      desired_size = var.desired_node_count

      disk_size = 20

      remote_access = {
          ec2_ssh_key = local.default_key_pair_name,
          source_security_group_ids = [ local.vpc_sg_id ]
      }

      iam_role_additional_policies = {
        AmazonEC2ContainerRegistryReadOnly = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
        AmazonSSMManagedInstanceCore       = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
      }

      tags = merge({ 
        Name = "${local.cluster_name}-ng-green"
      },local.common_tags)

      launch_template_tags = merge({ 
        Name = "${local.cluster_name}-ng-green"
      },local.common_tags)

      iam_role_tags = merge({ 
        Name = "${local.cluster_name}-ng-green"
      },local.common_tags)

    }
  }

  tags = local.common_tags
  node_security_group_tags = local.common_tags
  cluster_security_group_tags = local.common_tags

}

# EKS ADDONS 
module "eks_blueprints_addons" {
source = "aws-ia/eks-blueprints-addons/aws"
version = "~> 1.7.0" 

cluster_name      = module.eks.cluster_name
cluster_endpoint  = module.eks.cluster_endpoint
cluster_version   = module.eks.cluster_version
oidc_provider_arn = module.eks.oidc_provider

eks_addons = {
    aws-ebs-csi-driver = {
        most_recent = true
    }
    coredns = {
        most_recent = true
    }
    vpc-cni = {
        most_recent = true
    }
    kube-proxy = {
        most_recent = true
    }
}

enable_aws_load_balancer_controller    = true
enable_metrics_server                  = true
enable_cluster_autoscaler              = true

enable_cert_manager                    = true
cert_manager_route53_hosted_zone_arns  = ["arn:aws:route53:::hostedzone/${local.route53_hosted_zone_id}"]
}
bryantbiggs commented 1 year ago

you don't need to provide the exact configuration you are using, just a configuration that is both deployable and reproduces the issue you are seeing.

This looks like you are missing the IRSA role for the EBS CSI driver, but can't say definitively without a proper reproduction

jeenadeepak commented 1 year ago

@bryantbiggs : last day I have also tried IRSA as per example, also have fresh deployment with different type of configuration except argocd, every time I have faced same issue with cluster autoscaler, the above add-on source code I have shared is deployable.

bryantbiggs commented 1 year ago

unfortunately it is not deployable. Deployable means I can copy+paste and run terraform init && terraform apply

You can take a look at this example https://github.com/aws-ia/terraform-aws-eks-blueprints/tree/fa5d17d2d9444ff546c28e7d74be64deac45c2d6/examples/agones-game-controller or the examples here https://github.com/awslabs/data-on-eks for references where we use this module for deploying cluster autoscaler

jeenadeepak commented 1 year ago

@bryantbiggs : Kindly find the deployable code

terraform {
  required_version = ">= 1.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.13.1"
    }
    tls = {
      source = "hashicorp/tls"
      version = "~> 4.0.4"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.23.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 2.10.1"
    }
    kubectl = {
      source  = "gavinbunney/kubectl"
      version = "1.14.0"
    }
    http = {
      source  = "terraform-aws-modules/http"
      version = "2.4.1"
    }
  }

}

provider "aws" {
  region = local.region
}

data "aws_availability_zones" "available" {}

locals {
  name   = basename(path.cwd)
  region = "me-south-1"

  vpc_cidr = "10.0.0.0/16"
  azs      = slice(data.aws_availability_zones.available.names, 0, 3)

  tags = {
    Blueprint  = local.name
    GithubRepo = "github.com/aws-ia/terraform-aws-eks-blueprints"
  }
}

################################################################################
# Cluster
################################################################################

#tfsec:ignore:aws-eks-enable-control-plane-logging
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.13"

  cluster_name    = local.name
  cluster_version = "1.27"

  # EKS Addons
  cluster_endpoint_public_access  = true
  cluster_endpoint_private_access = true

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  eks_managed_node_groups = {
    initial = {
      capacity_type = "SPOT"
      instance_types = ["t3.xlarge"]

      min_size     = 1
      max_size     = 5
      desired_size = 3
    }
  }

  tags = local.tags
}
data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_name
  depends_on = [
    data.http.eks_cluster_readiness
  ]
}

data "http" "eks_cluster_readiness" {
  url            = join("/", [module.eks.cluster_endpoint, "healthz"])
  ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  timeout        = 600
}

provider "helm" {
  kubernetes {
    host                   = module.eks.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
    token                  = data.aws_eks_cluster_auth.cluster.token
  }
}
module "eks_blueprints_addons" {
  source = "aws-ia/eks-blueprints-addons/aws"
  version = "~> 1.7.0"

  cluster_name      = module.eks.cluster_name
  cluster_endpoint  = module.eks.cluster_endpoint
  cluster_version   = module.eks.cluster_version
  oidc_provider_arn = module.eks.oidc_provider

  eks_addons = {
    aws-ebs-csi-driver = {
      most_recent = true
    }
    coredns = {
      most_recent = true
    }
    vpc-cni = {
      most_recent = true
    }
    kube-proxy = {
      most_recent = true
    }
  }

  enable_aws_load_balancer_controller    = true
  enable_metrics_server                  = true
  enable_cluster_autoscaler              = true

}

################################################################################
# Supporting Resources
################################################################################

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  manage_default_vpc = true

  name = local.name
  cidr = local.vpc_cidr

  azs             = local.azs
  private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 4, k)]
  public_subnets  = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 48)]

  enable_dns_hostnames = true
  enable_dns_support   = true

  enable_nat_gateway = true
  single_nat_gateway =true

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = 1
  }

  tags = local.tags
}
JoeNorth commented 1 year ago

@jeenadeepak Please update oidc_provider_arn = module.eks.oidc_provider

to instead be

oidc_provider_arn = module.eks.oidc_provider_arn

This will fix the role trust policy and allow the pods to assume roles correctly.

bryantbiggs commented 1 year ago

thank you @JoeNorth - @jeenadeepak please see above to correct the issue

jeenadeepak commented 1 year ago

@bryantbiggs & @JoeNorth : Yes after the above changes it is working now, thanks