Closed ashishjullia closed 2 years ago
gjtempleton
commented
2 years ago @ashishjullia Can you provide instructions to reproduce this without needing external input (i.e. the EKS references here)?
When you say
Also, "autoDiscovery.clusterName" is always "null" do you mean after running the above Terraform, or that the default is null?
ashishjullia
commented
2 years ago @gjtempleton In order to reproduce this, you simply need to have an "eks cluster running" and "metrics server" installed. While I'm passing the following values:
set {
name = "rbac.serviceAccount.annotations.eks\\.amazonaws\\.com/role-arn"
value = module.cluster_autoscaler_irsa_role.iam_role_arn
}
set {
name = "deployment.annotations.cluster-autoscaler\\.kubernetes\\.io/safe-to-evict"
value = false
}
set {
name = "extraArgs.balance-similar-node-groups"
value = true
}
set {
name = "extraArgs.skip-nodes-with-system-pods"
value = true
}
set {
name = "image.tag"
value = "v${var.kubernetes_version}"
}All of the values are working completely fine but the only the following one is throwing the error mentioned above:
set {
name = "autoDiscovery.clusterName"
value = "${var.eks_cluster_name}"
}At the value part, I've tried the following:
Nothing is working for this particular variable's value.
I even tried the chart by directly modifying the "values.yaml" file still it is throwing the error on the same variable.
I'm almost kind of stuck here.
gjtempleton
commented
2 years ago Honestly, this doesn't seem like an issue with the chart itself to me, but a Terraform issue.
I don't use Terraform to manage my clusters or charts personally, so I don't know whether the data you're trying to pass in as the cluster name should exist or not.
ashishjullia
commented
2 years ago Oh, I see but seriously as far as my understanding goes, this is the problem with the chart itself.
Explanation -> I'm passing a string value and "autoDiscovery.clusterName" expects "string" value but the error is in relation with "bool".
Error: unable to build kubernetes objects from release manifest: unable to decode "": json: cannot unmarshal bool into Go struct field ObjectMeta.metadata.annotations of type string
│
│ with helm_release.cluster_autoscaler,
│ on helm.tf line 16, in resource "helm_release" "cluster_autoscaler":
│ 16: resource "helm_release" "cluster_autoscaler" {OK, if you still think this is a chart error, can you provide minimal steps to reproduce the error?
This is why we include the prompts for details in the template for bug reports which you've stripped out, I'm not familiar with Terraform, so to help you I need a way to reproduce this.
ashishjullia
commented
2 years ago @gjtempleton You don't need much to reproduce this but a k8s cluster is up and running (In my case I'm using eks) and the rest is to deploy the chart by passing a value to "autoDiscovery.clusterName".
I know this is not much information but apart from this there isn't anything else, I tried with both "terraform helm provider" as well as directly applied the chart via "helm install" by setting the above-mentioned values.
Why those values you might ask? Please take a look at this
gjtempleton
commented
2 years ago I'm sorry, but those aren't steps to reproduce this for someone who's unfamiliar with Terraform given I can tell I'll have to pass information on the cluster into Terraform somehow. I'm trying to help you here, but there's a limit to the amount of time I can spend in trying to find a way to reproduce it.
Are you saying that you get this same error with a pure helm apply now? Can you give me the values file that produces that error (appropriately redacted of anything sensitive)?
ashishjullia
commented
2 years ago @gjtempleton
Chart - nothing other than "values.yaml" is changed
My "values.yaml" is as follows:
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
# affinity -- Affinity for pod assignment
affinity: {}
autoDiscovery:
# cloudProviders `aws`, `gce`, `magnum` and `clusterapi` are supported by auto-discovery at this time
# AWS: Set tags as described in https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#auto-discovery-setup
# autoDiscovery.clusterName -- Enable autodiscovery for `cloudProvider=aws`, for groups matching `autoDiscovery.tags`.
# Enable autodiscovery for `cloudProvider=clusterapi`, for groups matching `autoDiscovery.labels`.
# Enable autodiscovery for `cloudProvider=gce`, but no MIG tagging required.
# Enable autodiscovery for `cloudProvider=magnum`, for groups matching `autoDiscovery.roles`.
clusterName: eks-cluster
# autoDiscovery.tags -- ASG tags to match, run through `tpl`.
tags:
- k8s.io/cluster-autoscaler/enabled
- k8s.io/cluster-autoscaler/eks-cluster
# - kubernetes.io/cluster/eks-cluster
# autoDiscovery.roles -- Magnum node group roles to match.
roles:
- worker
# autoDiscovery.labels -- Cluster-API labels to match https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/README.md#configuring-node-group-auto-discovery
labels: []
# - color: green
# - shape: circle
# autoscalingGroups -- For AWS, Azure AKS or Magnum. At least one element is required if not using `autoDiscovery`. For example:
# <pre>
# - name: asg1<br />
# maxSize: 2<br />
# minSize: 1
# </pre>
autoscalingGroups:
- name: eks-nodes-20220602105411243300000001-f8c09206-97e6-b8bc-d54f-4c702c467e46
maxSize: 6
minSize: 1
# - name: asg2
# maxSize: 2
# minSize: 1
# autoscalingGroupsnamePrefix -- For GCE. At least one element is required if not using `autoDiscovery`. For example:
# <pre>
# - name: ig01<br />
# maxSize: 10<br />
# minSize: 0
# </pre>
autoscalingGroupsnamePrefix: []
# - name: ig01
# maxSize: 10
# minSize: 0
# - name: ig02
# maxSize: 10
# minSize: 0
# awsAccessKeyID -- AWS access key ID ([if AWS user keys used](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#using-aws-credentials))
awsAccessKeyID: ""
# awsRegion -- AWS region (required if `cloudProvider=aws`)
awsRegion: us-east-1
# awsSecretAccessKey -- AWS access secret key ([if AWS user keys used](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#using-aws-credentials))
awsSecretAccessKey: ""
# azureClientID -- Service Principal ClientID with contributor permission to Cluster and Node ResourceGroup.
# Required if `cloudProvider=azure`
azureClientID: ""
# azureClientSecret -- Service Principal ClientSecret with contributor permission to Cluster and Node ResourceGroup.
# Required if `cloudProvider=azure`
azureClientSecret: ""
# azureResourceGroup -- Azure resource group that the cluster is located.
# Required if `cloudProvider=azure`
azureResourceGroup: ""
# azureSubscriptionID -- Azure subscription where the resources are located.
# Required if `cloudProvider=azure`
azureSubscriptionID: ""
# azureTenantID -- Azure tenant where the resources are located.
# Required if `cloudProvider=azure`
azureTenantID: ""
# azureVMType -- Azure VM type.
azureVMType: "AKS"
# azureClusterName -- Azure AKS cluster name.
# Required if `cloudProvider=azure`
azureClusterName: ""
# azureNodeResourceGroup -- Azure resource group where the cluster's nodes are located, typically set as `MC_<cluster-resource-group-name>_<cluster-name>_<location>`.
# Required if `cloudProvider=azure`
azureNodeResourceGroup: ""
# azureUseManagedIdentityExtension -- Whether to use Azure's managed identity extension for credentials. If using MSI, ensure subscription ID, resource group, and azure AKS cluster name are set.
azureUseManagedIdentityExtension: false
# magnumClusterName -- Cluster name or ID in Magnum.
# Required if `cloudProvider=magnum` and not setting `autoDiscovery.clusterName`.
magnumClusterName: ""
# magnumCABundlePath -- Path to the host's CA bundle, from `ca-file` in the cloud-config file.
magnumCABundlePath: "/etc/kubernetes/ca-bundle.crt"
# clusterAPIMode -- Cluster API mode, see https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/README.md#connecting-cluster-autoscaler-to-cluster-api-management-and-workload-clusters
# Syntax: workloadClusterMode-ManagementClusterMode
# for `kubeconfig-kubeconfig`, `incluster-kubeconfig` and `single-kubeconfig` you always must mount the external kubeconfig using either `extraVolumeSecrets` or `extraMounts` and `extraVolumes`
# if you dont set `clusterAPIKubeconfigSecret`and thus use an in-cluster config or want to use a non capi generated kubeconfig you must do so for the workload kubeconfig as well
clusterAPIMode: incluster-incluster # incluster-incluster, incluster-kubeconfig, kubeconfig-incluster, kubeconfig-kubeconfig, single-kubeconfig
# clusterAPIKubeconfigSecret -- Secret containing kubeconfig for connecting to Cluster API managed workloadcluster
# Required if `cloudProvider=clusterapi` and `clusterAPIMode=kubeconfig-kubeconfig,kubeconfig-incluster or incluster-kubeconfig`
clusterAPIKubeconfigSecret: ""
# clusterAPIWorkloadKubeconfigPath -- Path to kubeconfig for connecting to Cluster API managed workloadcluster, only used if `clusterAPIMode=kubeconfig-kubeconfig or kubeconfig-incluster`
clusterAPIWorkloadKubeconfigPath: /etc/kubernetes/value
# clusterAPICloudConfigPath -- Path to kubeconfig for connecting to Cluster API Management Cluster, only used if `clusterAPIMode=kubeconfig-kubeconfig or incluster-kubeconfig`
clusterAPICloudConfigPath: /etc/kubernetes/mgmt-kubeconfig
# clusterAPIConfigMapsNamespace -- Namespace on the workload cluster to store Leader election and status configmaps
clusterAPIConfigMapsNamespace: ""
# cloudConfigPath -- Configuration file for cloud provider.
cloudConfigPath: /etc/gce.conf
# cloudProvider -- The cloud provider where the autoscaler runs.
# Currently only `gce`, `aws`, `azure`, `magnum` and `clusterapi` are supported.
# `aws` supported for AWS. `gce` for GCE. `azure` for Azure AKS.
# `magnum` for OpenStack Magnum, `clusterapi` for Cluster API.
cloudProvider: aws
# containerSecurityContext -- [Security context for container](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)
containerSecurityContext: {}
# capabilities:
# drop:
# - ALL
deployment:
# deployment.annotations -- Annotations to add to the Deployment object.
annotations: {
cluster-autoscaler.kubernetes.io/safe-to-evict: false
}
# dnsPolicy -- Defaults to `ClusterFirst`. Valid values are:
# `ClusterFirstWithHostNet`, `ClusterFirst`, `Default` or `None`.
# If autoscaler does not depend on cluster DNS, recommended to set this to `Default`.
dnsPolicy: ClusterFirst
## Priorities Expander
# expanderPriorities -- The expanderPriorities is used if `extraArgs.expander` contains `priority` and expanderPriorities is also set with the priorities.
# If `extraArgs.expander` contains `priority`, then expanderPriorities is used to define cluster-autoscaler-priority-expander priorities.
# See: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/expander/priority/readme.md
expanderPriorities: {}
# priorityConfigMapAnnotations -- Annotations to add to `cluster-autoscaler-priority-expander` ConfigMap.
priorityConfigMapAnnotations: {}
# key1: "value1"
# key2: "value2"
# extraArgs -- Additional container arguments.
# Refer to https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-are-the-parameters-to-ca for the full list of cluster autoscaler
# parameters and their default values.
# Everything after the first _ will be ignored allowing the use of multi-string arguments.
extraArgs:
logtostderr: true
stderrthreshold: info
v: 4
# write-status-configmap: true
# status-config-map-name: cluster-autoscaler-status
# leader-elect: true
# leader-elect-resource-lock: endpoints
# skip-nodes-with-local-storage: true
# expander: random
# scale-down-enabled: true
balance-similar-node-groups: true
# min-replica-count: 0
# scale-down-utilization-threshold: 0.5
# scale-down-non-empty-candidates-count: 30
# max-node-provision-time: 15m0s
# scan-interval: 10s
# scale-down-delay-after-add: 10m
# scale-down-delay-after-delete: 0s
# scale-down-delay-after-failure: 3m
# scale-down-unneeded-time: 10m
skip-nodes-with-system-pods: true
# balancing-ignore-label_1: first-label-to-ignore
# balancing-ignore-label_2: second-label-to-ignore
# extraEnv -- Additional container environment variables.
extraEnv: {}
# extraEnvConfigMaps -- Additional container environment variables from ConfigMaps.
extraEnvConfigMaps: {}
# extraEnvSecrets -- Additional container environment variables from Secrets.
extraEnvSecrets: {}
# envFromConfigMap -- ConfigMap name to use as envFrom.
envFromConfigMap: ""
# envFromSecret -- Secret name to use as envFrom.
envFromSecret: ""
# extraVolumeSecrets -- Additional volumes to mount from Secrets.
extraVolumeSecrets: {}
# autoscaler-vol:
# mountPath: /data/autoscaler/
# custom-vol:
# name: custom-secret
# mountPath: /data/custom/
# items:
# - key: subkey
# path: mypath
# extraVolumes -- Additional volumes.
extraVolumes: []
# - name: ssl-certs
# hostPath:
# path: /etc/ssl/certs/ca-bundle.crt
# extraVolumeMounts -- Additional volumes to mount.
extraVolumeMounts: []
# - name: ssl-certs
# mountPath: /etc/ssl/certs/ca-certificates.crt
# readOnly: true
# fullnameOverride -- String to fully override `cluster-autoscaler.fullname` template.
fullnameOverride: ""
image:
# image.repository -- Image repository
repository: k8s.gcr.io/autoscaling/cluster-autoscaler
# image.tag -- Image tag
tag: v1.22.0
# image.pullPolicy -- Image pull policy
pullPolicy: IfNotPresent
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
# image.pullSecrets -- Image pull secrets
pullSecrets: []
# - myRegistrKeySecretName
# kubeTargetVersionOverride -- Allow overriding the `.Capabilities.KubeVersion.GitVersion` check. Useful for `helm template` commands.
kubeTargetVersionOverride: ""
# nameOverride -- String to partially override `cluster-autoscaler.fullname` template (will maintain the release name)
nameOverride: ""
# nodeSelector -- Node labels for pod assignment. Ref: https://kubernetes.io/docs/user-guide/node-selection/.
nodeSelector: {}
# podAnnotations -- Annotations to add to each pod.
podAnnotations: {}
# podDisruptionBudget -- Pod disruption budget.
podDisruptionBudget:
maxUnavailable: 1
# minAvailable: 2
# podLabels -- Labels to add to each pod.
podLabels: {}
# additionalLabels -- Labels to add to each object of the chart.
additionalLabels: {}
# priorityClassName -- priorityClassName
priorityClassName: "system-cluster-critical"
rbac:
# rbac.create -- If `true`, create and use RBAC resources.
create: true
# rbac.pspEnabled -- If `true`, creates and uses RBAC resources required in the cluster with [Pod Security Policies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/) enabled.
# Must be used with `rbac.create` set to `true`.
pspEnabled: false
# rbac.clusterScoped -- if set to false will only provision RBAC to alter resources in the current namespace. Most useful for Cluster-API
clusterScoped: true
serviceAccount:
# rbac.serviceAccount.annotations -- Additional Service Account annotations.
annotations: {
eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxxxxx:role/cluster-autoscaler
}
# rbac.serviceAccount.create -- If `true` and `rbac.create` is also true, a Service Account will be created.
create: true
# rbac.serviceAccount.name -- The name of the ServiceAccount to use. If not set and create is `true`, a name is generated using the fullname template.
name: ""
# rbac.serviceAccount.automountServiceAccountToken -- Automount API credentials for a Service Account.
automountServiceAccountToken: true
# replicaCount -- Desired number of pods
replicaCount: 1
# resources -- Pod resource requests and limits.
resources: {}
# limits:
# cpu: 100m
# memory: 300Mi
# requests:
# cpu: 100m
# memory: 300Mi
# securityContext -- [Security context for pod](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)
securityContext: {}
# runAsNonRoot: true
# runAsUser: 1001
# runAsGroup: 1001
service:
# service.create -- If `true`, a Service will be created.
create: true
# service.annotations -- Annotations to add to service
annotations: {}
# service.labels -- Labels to add to service
labels: {}
# service.externalIPs -- List of IP addresses at which the service is available. Ref: https://kubernetes.io/docs/user-guide/services/#external-ips.
externalIPs: []
# service.loadBalancerIP -- IP address to assign to load balancer (if supported).
loadBalancerIP: ""
# service.loadBalancerSourceRanges -- List of IP CIDRs allowed access to load balancer (if supported).
loadBalancerSourceRanges: []
# service.servicePort -- Service port to expose.
servicePort: 8085
# service.portName -- Name for service port.
portName: http
# service.type -- Type of service to create.
type: ClusterIP
## Are you using Prometheus Operator?
serviceMonitor:
# serviceMonitor.enabled -- If true, creates a Prometheus Operator ServiceMonitor.
enabled: false
# serviceMonitor.interval -- Interval that Prometheus scrapes Cluster Autoscaler metrics.
interval: 10s
# serviceMonitor.namespace -- Namespace which Prometheus is running in.
namespace: monitoring
## [Prometheus Selector Label](https://github.com/helm/charts/tree/master/stable/prometheus-operator#prometheus-operator-1)
## [Kube Prometheus Selector Label](https://github.com/helm/charts/tree/master/stable/prometheus-operator#exporters)
# serviceMonitor.selector -- Default to kube-prometheus install (CoreOS recommended), but should be set according to Prometheus install.
selector:
release: prometheus-operator
# serviceMonitor.path -- The path to scrape for metrics; autoscaler exposes `/metrics` (this is standard)
path: /metrics
## Custom PrometheusRule to be defined
## The value is evaluated as a template, so, for example, the value can depend on .Release or .Chart
## ref: https://github.com/coreos/prometheus-operator#customresourcedefinitions
prometheusRule:
# prometheusRule.enabled -- If true, creates a Prometheus Operator PrometheusRule.
enabled: false
# prometheusRule.additionalLabels -- Additional labels to be set in metadata.
additionalLabels: {}
# prometheusRule.namespace -- Namespace which Prometheus is running in.
namespace: monitoring
# prometheusRule.interval -- How often rules in the group are evaluated (falls back to `global.evaluation_interval` if not set).
interval: null
# prometheusRule.rules -- Rules spec template (see https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#rule).
rules: []
# tolerations -- List of node taints to tolerate (requires Kubernetes >= 1.6).
tolerations: []
# topologySpreadConstraints -- You can use topology spread constraints to control how Pods are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains. (requires Kubernetes >= 1.19).
topologySpreadConstraints: []
# - maxSkew: 1
# topologyKey: topology.kubernetes.io/zone
# whenUnsatisfiable: DoNotSchedule
# labelSelector:
# matchLabels:
# app.kubernetes.io/instance: cluster-autoscaler
# updateStrategy -- [Deployment update strategy](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy)
updateStrategy: {}
# rollingUpdate:
# maxSurge: 1
# maxUnavailable: 0
# type: RollingUpdate
helm install my-release autoscaler/charts/cluster-autoscaler/
Following error occurs:
Error: unable to build kubernetes objects from release manifest: unable to decode "": json: cannot unmarshal bool into Go struct field ObjectMeta.metadata.annotations of type string
And when I comment clusterName: eks-cluster - no such error
gjtempleton
commented
2 years ago And what version of the chart and helm is this with (as I can successfully template the chart with that values file)?
ashishjullia
commented
2 years ago Yeah, here it is -> "9.19.0" - latest one
gjtempleton
commented
2 years ago And what version of helm itself? That file works fine for me.
ashishjullia
commented
2 years ago version.BuildInfo{Version:"v3.8.0", GitCommit:"d14138609b01886f544b2025f5000351c9eb092e", GitTreeState:"clean", GoVersion:"go1.17.5"}
@gjtempleton also, how you actually verified that the file works for you and I'm assuming that you haven't commented clusterName: eks-cluster?
gjtempleton
commented
2 years ago I was running helm template, which works fine even with your example values file. I've now spun up a local cluster and reproduced your error, it only appears with helm install.
The issue is not the clusterName, it only makes the error appear as the deployment isn't created without that variable. If you diff your values.yaml file, against the defaults, you'll see an annotation, trying to use a bool, which maps to the error message:
deployment:
# deployment.annotations -- Annotations to add to the Deployment object.
annotations: {
cluster-autoscaler.kubernetes.io/safe-to-evict: false
}@gjtempleton oh okay, one more thing, how to actually replicate this, as this is what I'm trying to achieve here.
gjtempleton
commented
2 years ago You need to quote the false as a string, as the example shows.
ashishjullia
commented
2 years ago @gjtempleton Yes I tried that before posting the most recent comment but same error.
gjtempleton
commented
2 years ago What exactly did you try, because I tried with exactly the same
values.yaml, only with "false" rather than false, and didn't get the
error.
ashishjullia
commented
2 years ago @gjtempleton I used the following:
deployment:
# deployment.annotations -- Annotations to add to the Deployment object.
annotations: {
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
}The below values file should only differ from the example you gave in having the quotes I suggested on the annotation.
It works for me when running helm install -v 4 my-release . in the chart directory.
# - name: asg1autoscalingGroups: - name: eks-nodes-20220602105411243300000001-f8c09206-97e6-b8bc-d54f-4c702c467e46 maxSize: 6 minSize: 1 # - name: asg2 # maxSize: 2 # minSize: 1 # autoscalingGroupsnamePrefix -- For GCE. At least one element is required if not using `autoDiscovery`. For example: #
# maxSize: 2
# minSize: 1 #
# - name: ig01autoscalingGroupsnamePrefix: [] # - name: ig01 # maxSize: 10 # minSize: 0 # - name: ig02 # maxSize: 10 # minSize: 0 # awsAccessKeyID -- AWS access key ID ([if AWS user keys used](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#using-aws-credentials)) awsAccessKeyID: "" # awsRegion -- AWS region (required if `cloudProvider=aws`) awsRegion: us-east-1 # awsSecretAccessKey -- AWS access secret key ([if AWS user keys used](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#using-aws-credentials)) awsSecretAccessKey: "" # azureClientID -- Service Principal ClientID with contributor permission to Cluster and Node ResourceGroup. # Required if `cloudProvider=azure` azureClientID: "" # azureClientSecret -- Service Principal ClientSecret with contributor permission to Cluster and Node ResourceGroup. # Required if `cloudProvider=azure` azureClientSecret: "" # azureResourceGroup -- Azure resource group that the cluster is located. # Required if `cloudProvider=azure` azureResourceGroup: "" # azureSubscriptionID -- Azure subscription where the resources are located. # Required if `cloudProvider=azure` azureSubscriptionID: "" # azureTenantID -- Azure tenant where the resources are located. # Required if `cloudProvider=azure` azureTenantID: "" # azureVMType -- Azure VM type. azureVMType: "AKS" # azureClusterName -- Azure AKS cluster name. # Required if `cloudProvider=azure` azureClusterName: "" # azureNodeResourceGroup -- Azure resource group where the cluster's nodes are located, typically set as `MC_
# maxSize: 10
# minSize: 0 #
ashishjullia
commented
2 years ago @gjtempleton You pinpointing "false" helped me solve that error but there is still one problem that is affecting the process of "cluster autoscaler" deployment via "helm chart".
When I'm applying the following "yaml" to the cluster, everything is working fine:
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
name: cluster-autoscaler
namespace: kube-system
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxx:role/cluster-autoscaler
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cluster-autoscaler
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
rules:
- apiGroups: [""]
resources: ["events", "endpoints"]
verbs: ["create", "patch"]
- apiGroups: [""]
resources: ["pods/eviction"]
verbs: ["create"]
- apiGroups: [""]
resources: ["pods/status"]
verbs: ["update"]
- apiGroups: [""]
resources: ["endpoints"]
resourceNames: ["cluster-autoscaler"]
verbs: ["get", "update"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["watch", "list", "get", "update"]
- apiGroups: [""]
resources:
- "namespaces"
- "pods"
- "services"
- "replicationcontrollers"
- "persistentvolumeclaims"
- "persistentvolumes"
verbs: ["watch", "list", "get"]
- apiGroups: ["extensions"]
resources: ["replicasets", "daemonsets"]
verbs: ["watch", "list", "get"]
- apiGroups: ["policy"]
resources: ["poddisruptionbudgets"]
verbs: ["watch", "list"]
- apiGroups: ["apps"]
resources: ["statefulsets", "replicasets", "daemonsets"]
verbs: ["watch", "list", "get"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses", "csinodes", "csidrivers", "csistoragecapacities"]
verbs: ["watch", "list", "get"]
- apiGroups: ["batch", "extensions"]
resources: ["jobs"]
verbs: ["get", "list", "watch", "patch"]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["create"]
- apiGroups: ["coordination.k8s.io"]
resourceNames: ["cluster-autoscaler"]
resources: ["leases"]
verbs: ["get", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: cluster-autoscaler
namespace: kube-system
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["create","list","watch"]
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["cluster-autoscaler-status", "cluster-autoscaler-priority-expander"]
verbs: ["delete", "get", "update", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cluster-autoscaler
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-autoscaler
subjects:
- kind: ServiceAccount
name: cluster-autoscaler
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: cluster-autoscaler
namespace: kube-system
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: cluster-autoscaler
subjects:
- kind: ServiceAccount
name: cluster-autoscaler
namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: cluster-autoscaler
namespace: kube-system
labels:
app: cluster-autoscaler
spec:
replicas: 1
selector:
matchLabels:
app: cluster-autoscaler
template:
metadata:
labels:
app: cluster-autoscaler
annotations:
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
prometheus.io/scrape: 'true'
prometheus.io/port: '8085'
spec:
priorityClassName: system-cluster-critical
securityContext:
runAsNonRoot: true
runAsUser: 65534
fsGroup: 65534
serviceAccountName: cluster-autoscaler
containers:
- image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.22.2
name: cluster-autoscaler
resources:
limits:
cpu: 100m
memory: 600Mi
requests:
cpu: 100m
memory: 600Mi
command:
- ./cluster-autoscaler
- --v=4
- --stderrthreshold=info
- --cloud-provider=aws
- --skip-nodes-with-local-storage=false
- --expander=least-waste
- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/eks-cluster
- --balance-similar-node-groups
- --skip-nodes-with-system-pods=false
volumeMounts:
- name: ssl-certs
mountPath: /etc/ssl/certs/ca-certificates.crt #/etc/ssl/certs/ca-bundle.crt for Amazon Linux Worker Nodes
readOnly: true
imagePullPolicy: "Always"
volumes:
- name: ssl-certs
hostPath:
path: "/etc/ssl/certs/ca-bundle.crt"
But when trying to replicate the same with the following values in the "values.yaml" from chart:
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
# affinity -- Affinity for pod assignment
affinity: {}
autoDiscovery:
# cloudProviders `aws`, `gce`, `magnum` and `clusterapi` are supported by auto-discovery at this time
# AWS: Set tags as described in https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#auto-discovery-setup
# autoDiscovery.clusterName -- Enable autodiscovery for `cloudProvider=aws`, for groups matching `autoDiscovery.tags`.
# Enable autodiscovery for `cloudProvider=clusterapi`, for groups matching `autoDiscovery.labels`.
# Enable autodiscovery for `cloudProvider=gce`, but no MIG tagging required.
# Enable autodiscovery for `cloudProvider=magnum`, for groups matching `autoDiscovery.roles`.
clusterName: eks-cluster
# autoDiscovery.tags -- ASG tags to match, run through `tpl`.
tags:
- k8s.io/cluster-autoscaler/enabled
- k8s.io/cluster-autoscaler/{{ .Values.autoDiscovery.clusterName }}
# - kubernetes.io/cluster/eks-cluster
# autoDiscovery.roles -- Magnum node group roles to match.
roles:
- worker
# autoDiscovery.labels -- Cluster-API labels to match https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/README.md#configuring-node-group-auto-discovery
labels: []
# - color: green
# - shape: circle
# autoscalingGroups -- For AWS, Azure AKS or Magnum. At least one element is required if not using `autoDiscovery`. For example:
# <pre>
# - name: asg1<br />
# maxSize: 2<br />
# minSize: 1
# </pre>
autoscalingGroups: []
# - name: eks-nodes-20220602105411243300000001-f8c09206-97e6-b8bc-d54f-4c702c467e46
# maxSize: 6
# minSize: 1
# - name: asg2
# maxSize: 2
# minSize: 1
# autoscalingGroupsnamePrefix -- For GCE. At least one element is required if not using `autoDiscovery`. For example:
# <pre>
# - name: ig01<br />
# maxSize: 10<br />
# minSize: 0
# </pre>
autoscalingGroupsnamePrefix: []
# - name: ig01
# maxSize: 10
# minSize: 0
# - name: ig02
# maxSize: 10
# minSize: 0
# awsAccessKeyID -- AWS access key ID ([if AWS user keys used](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#using-aws-credentials))
awsAccessKeyID: ""
# awsRegion -- AWS region (required if `cloudProvider=aws`)
awsRegion: us-east-1
# awsSecretAccessKey -- AWS access secret key ([if AWS user keys used](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#using-aws-credentials))
awsSecretAccessKey: ""
# azureClientID -- Service Principal ClientID with contributor permission to Cluster and Node ResourceGroup.
# Required if `cloudProvider=azure`
azureClientID: ""
# azureClientSecret -- Service Principal ClientSecret with contributor permission to Cluster and Node ResourceGroup.
# Required if `cloudProvider=azure`
azureClientSecret: ""
# azureResourceGroup -- Azure resource group that the cluster is located.
# Required if `cloudProvider=azure`
azureResourceGroup: ""
# azureSubscriptionID -- Azure subscription where the resources are located.
# Required if `cloudProvider=azure`
azureSubscriptionID: ""
# azureTenantID -- Azure tenant where the resources are located.
# Required if `cloudProvider=azure`
azureTenantID: ""
# azureVMType -- Azure VM type.
azureVMType: "AKS"
# azureClusterName -- Azure AKS cluster name.
# Required if `cloudProvider=azure`
azureClusterName: ""
# azureNodeResourceGroup -- Azure resource group where the cluster's nodes are located, typically set as `MC_<cluster-resource-group-name>_<cluster-name>_<location>`.
# Required if `cloudProvider=azure`
azureNodeResourceGroup: ""
# azureUseManagedIdentityExtension -- Whether to use Azure's managed identity extension for credentials. If using MSI, ensure subscription ID, resource group, and azure AKS cluster name are set.
azureUseManagedIdentityExtension: false
# magnumClusterName -- Cluster name or ID in Magnum.
# Required if `cloudProvider=magnum` and not setting `autoDiscovery.clusterName`.
magnumClusterName: ""
# magnumCABundlePath -- Path to the host's CA bundle, from `ca-file` in the cloud-config file.
magnumCABundlePath: "/etc/kubernetes/ca-bundle.crt"
# clusterAPIMode -- Cluster API mode, see https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/README.md#connecting-cluster-autoscaler-to-cluster-api-management-and-workload-clusters
# Syntax: workloadClusterMode-ManagementClusterMode
# for `kubeconfig-kubeconfig`, `incluster-kubeconfig` and `single-kubeconfig` you always must mount the external kubeconfig using either `extraVolumeSecrets` or `extraMounts` and `extraVolumes`
# if you dont set `clusterAPIKubeconfigSecret`and thus use an in-cluster config or want to use a non capi generated kubeconfig you must do so for the workload kubeconfig as well
clusterAPIMode: incluster-incluster # incluster-incluster, incluster-kubeconfig, kubeconfig-incluster, kubeconfig-kubeconfig, single-kubeconfig
# clusterAPIKubeconfigSecret -- Secret containing kubeconfig for connecting to Cluster API managed workloadcluster
# Required if `cloudProvider=clusterapi` and `clusterAPIMode=kubeconfig-kubeconfig,kubeconfig-incluster or incluster-kubeconfig`
clusterAPIKubeconfigSecret: ""
# clusterAPIWorkloadKubeconfigPath -- Path to kubeconfig for connecting to Cluster API managed workloadcluster, only used if `clusterAPIMode=kubeconfig-kubeconfig or kubeconfig-incluster`
clusterAPIWorkloadKubeconfigPath: /etc/kubernetes/value
# clusterAPICloudConfigPath -- Path to kubeconfig for connecting to Cluster API Management Cluster, only used if `clusterAPIMode=kubeconfig-kubeconfig or incluster-kubeconfig`
clusterAPICloudConfigPath: /etc/kubernetes/mgmt-kubeconfig
# clusterAPIConfigMapsNamespace -- Namespace on the workload cluster to store Leader election and status configmaps
clusterAPIConfigMapsNamespace: ""
# cloudConfigPath -- Configuration file for cloud provider.
cloudConfigPath: /etc/gce.conf
# cloudProvider -- The cloud provider where the autoscaler runs.
# Currently only `gce`, `aws`, `azure`, `magnum` and `clusterapi` are supported.
# `aws` supported for AWS. `gce` for GCE. `azure` for Azure AKS.
# `magnum` for OpenStack Magnum, `clusterapi` for Cluster API.
cloudProvider: aws
# containerSecurityContext -- [Security context for container](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)
containerSecurityContext: {}
# capabilities:
# drop:
# - ALL
deployment:
# deployment.annotations -- Annotations to add to the Deployment object.
annotations:
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
# dnsPolicy -- Defaults to `ClusterFirst`. Valid values are:
# `ClusterFirstWithHostNet`, `ClusterFirst`, `Default` or `None`.
# If autoscaler does not depend on cluster DNS, recommended to set this to `Default`.
dnsPolicy: ClusterFirst
## Priorities Expander
# expanderPriorities -- The expanderPriorities is used if `extraArgs.expander` contains `priority` and expanderPriorities is also set with the priorities.
# If `extraArgs.expander` contains `priority`, then expanderPriorities is used to define cluster-autoscaler-priority-expander priorities.
# See: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/expander/priority/readme.md
expanderPriorities: {}
# priorityConfigMapAnnotations -- Annotations to add to `cluster-autoscaler-priority-expander` ConfigMap.
priorityConfigMapAnnotations: {}
# key1: "value1"
# key2: "value2"
# extraArgs -- Additional container arguments.
# Refer to https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-are-the-parameters-to-ca for the full list of cluster autoscaler
# parameters and their default values.
# Everything after the first _ will be ignored allowing the use of multi-string arguments.
extraArgs:
logtostderr: true
stderrthreshold: info
v: 4
# write-status-configmap: true
# status-config-map-name: cluster-autoscaler-status
# leader-elect: true
# leader-elect-resource-lock: endpoints
skip-nodes-with-local-storage: false
expander: least-waste
# scale-down-enabled: true
balance-similar-node-groups: true
# min-replica-count: 0
# scale-down-utilization-threshold: 0.5
# scale-down-non-empty-candidates-count: 30
# max-node-provision-time: 15m0s
# scan-interval: 10s
# scale-down-delay-after-add: 10m
# scale-down-delay-after-delete: 0s
# scale-down-delay-after-failure: 3m
# scale-down-unneeded-time: 10m
skip-nodes-with-system-pods: true
# balancing-ignore-label_1: first-label-to-ignore
# balancing-ignore-label_2: second-label-to-ignore
# extraEnv -- Additional container environment variables.
extraEnv: {}
# extraEnvConfigMaps -- Additional container environment variables from ConfigMaps.
extraEnvConfigMaps: {}
# extraEnvSecrets -- Additional container environment variables from Secrets.
extraEnvSecrets: {}
# envFromConfigMap -- ConfigMap name to use as envFrom.
envFromConfigMap: ""
# envFromSecret -- Secret name to use as envFrom.
envFromSecret: ""
# extraVolumeSecrets -- Additional volumes to mount from Secrets.
extraVolumeSecrets: {}
# autoscaler-vol:
# mountPath: /data/autoscaler/
# custom-vol:
# name: custom-secret
# mountPath: /data/custom/
# items:
# - key: subkey
# path: mypath
# extraVolumes -- Additional volumes.
extraVolumes:
- name: ssl-certs
hostPath:
path: /etc/ssl/certs/ca-bundle.crt
# extraVolumeMounts -- Additional volumes to mount.
extraVolumeMounts:
- name: ssl-certs
mountPath: /etc/ssl/certs/ca-certificates.crt
readOnly: true
# fullnameOverride -- String to fully override `cluster-autoscaler.fullname` template.
fullnameOverride: ""
image:
# image.repository -- Image repository
repository: k8s.gcr.io/autoscaling/cluster-autoscaler
# image.tag -- Image tag
tag: v1.22.0
# image.pullPolicy -- Image pull policy
pullPolicy: IfNotPresent
## Optionally specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
# image.pullSecrets -- Image pull secrets
pullSecrets: []
# - myRegistrKeySecretName
# kubeTargetVersionOverride -- Allow overriding the `.Capabilities.KubeVersion.GitVersion` check. Useful for `helm template` commands.
kubeTargetVersionOverride: ""
# nameOverride -- String to partially override `cluster-autoscaler.fullname` template (will maintain the release name)
nameOverride: ""
# nodeSelector -- Node labels for pod assignment. Ref: https://kubernetes.io/docs/user-guide/node-selection/.
nodeSelector: {}
# podAnnotations -- Annotations to add to each pod.
podAnnotations: {}
# podDisruptionBudget -- Pod disruption budget.
podDisruptionBudget:
maxUnavailable: 1
# minAvailable: 2
# podLabels -- Labels to add to each pod.
podLabels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
# additionalLabels -- Labels to add to each object of the chart.
additionalLabels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
# priorityClassName -- priorityClassName
priorityClassName: "system-cluster-critical"
rbac:
# rbac.create -- If `true`, create and use RBAC resources.
create: true
# rbac.pspEnabled -- If `true`, creates and uses RBAC resources required in the cluster with [Pod Security Policies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/) enabled.
# Must be used with `rbac.create` set to `true`.
pspEnabled: false
# rbac.clusterScoped -- if set to false will only provision RBAC to alter resources in the current namespace. Most useful for Cluster-API
clusterScoped: true
serviceAccount:
# rbac.serviceAccount.annotations -- Additional Service Account annotations.
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::xxxxxxxxxxx:role/cluster-autoscaler"
# rbac.serviceAccount.create -- If `true` and `rbac.create` is also true, a Service Account will be created.
create: true
# rbac.serviceAccount.name -- The name of the ServiceAccount to use. If not set and create is `true`, a name is generated using the fullname template.
name: "cluster-autoscaler"
# rbac.serviceAccount.automountServiceAccountToken -- Automount API credentials for a Service Account.
automountServiceAccountToken: true
# replicaCount -- Desired number of pods
replicaCount: 1
# resources -- Pod resource requests and limits.
resources:
limits:
cpu: 100m
memory: 600Mi
requests:
cpu: 100m
memory: 600Mi
# securityContext -- [Security context for pod](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)
securityContext:
runAsNonRoot: true
runAsUser: 65534
runAsGroup: 65534
service:
# service.create -- If `true`, a Service will be created.
create: true
# service.annotations -- Annotations to add to service
annotations: {}
# service.labels -- Labels to add to service
labels: {}
# service.externalIPs -- List of IP addresses at which the service is available. Ref: https://kubernetes.io/docs/user-guide/services/#external-ips.
externalIPs: []
# service.loadBalancerIP -- IP address to assign to load balancer (if supported).
loadBalancerIP: ""
# service.loadBalancerSourceRanges -- List of IP CIDRs allowed access to load balancer (if supported).
loadBalancerSourceRanges: []
# service.servicePort -- Service port to expose.
servicePort: 8085
# service.portName -- Name for service port.
portName: http
# service.type -- Type of service to create.
type: ClusterIP
## Are you using Prometheus Operator?
serviceMonitor:
# serviceMonitor.enabled -- If true, creates a Prometheus Operator ServiceMonitor.
enabled: false
# serviceMonitor.interval -- Interval that Prometheus scrapes Cluster Autoscaler metrics.
interval: 10s
# serviceMonitor.namespace -- Namespace which Prometheus is running in.
namespace: monitoring
## [Prometheus Selector Label](https://github.com/helm/charts/tree/master/stable/prometheus-operator#prometheus-operator-1)
## [Kube Prometheus Selector Label](https://github.com/helm/charts/tree/master/stable/prometheus-operator#exporters)
# serviceMonitor.selector -- Default to kube-prometheus install (CoreOS recommended), but should be set according to Prometheus install.
selector:
release: prometheus-operator
# serviceMonitor.path -- The path to scrape for metrics; autoscaler exposes `/metrics` (this is standard)
path: /metrics
## Custom PrometheusRule to be defined
## The value is evaluated as a template, so, for example, the value can depend on .Release or .Chart
## ref: https://github.com/coreos/prometheus-operator#customresourcedefinitions
prometheusRule:
# prometheusRule.enabled -- If true, creates a Prometheus Operator PrometheusRule.
enabled: false
# prometheusRule.additionalLabels -- Additional labels to be set in metadata.
additionalLabels: {}
# prometheusRule.namespace -- Namespace which Prometheus is running in.
namespace: monitoring
# prometheusRule.interval -- How often rules in the group are evaluated (falls back to `global.evaluation_interval` if not set).
interval: null
# prometheusRule.rules -- Rules spec template (see https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#rule).
rules: []
# tolerations -- List of node taints to tolerate (requires Kubernetes >= 1.6).
tolerations: []
# topologySpreadConstraints -- You can use topology spread constraints to control how Pods are spread across your cluster among failure-domains such as regions, zones, nodes, and other user-defined topology domains. (requires Kubernetes >= 1.19).
topologySpreadConstraints: []
# - maxSkew: 1
# topologyKey: topology.kubernetes.io/zone
# whenUnsatisfiable: DoNotSchedule
# labelSelector:
# matchLabels:
# app.kubernetes.io/instance: cluster-autoscaler
# updateStrategy -- [Deployment update strategy](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy)
updateStrategy: {}
# rollingUpdate:
# maxSurge: 1
# maxUnavailable: 0
# type: RollingUpdate
I'm getting the following error on pod level:
❯ kubectl logs my-release-aws-cluster-autoscaler-c4fb847fb-r9wm9
I0603 12:59:25.224229 1 flags.go:52] FLAG: --add-dir-header="false"
I0603 12:59:25.224279 1 flags.go:52] FLAG: --address=":8085"
I0603 12:59:25.224284 1 flags.go:52] FLAG: --alsologtostderr="false"
I0603 12:59:25.224289 1 flags.go:52] FLAG: --aws-use-static-instance-list="false"
I0603 12:59:25.224293 1 flags.go:52] FLAG: --balance-similar-node-groups="true"
I0603 12:59:25.224298 1 flags.go:52] FLAG: --balancing-ignore-label="[]"
I0603 12:59:25.224302 1 flags.go:52] FLAG: --cloud-config=""
I0603 12:59:25.224308 1 flags.go:52] FLAG: --cloud-provider="aws"
I0603 12:59:25.224312 1 flags.go:52] FLAG: --cloud-provider-gce-l7lb-src-cidrs="130.211.0.0/22,35.191.0.0/16"
I0603 12:59:25.224319 1 flags.go:52] FLAG: --cloud-provider-gce-lb-src-cidrs="130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16"
I0603 12:59:25.224326 1 flags.go:52] FLAG: --cluster-name=""
I0603 12:59:25.224331 1 flags.go:52] FLAG: --clusterapi-cloud-config-authoritative="false"
I0603 12:59:25.224336 1 flags.go:52] FLAG: --cordon-node-before-terminating="false"
I0603 12:59:25.224341 1 flags.go:52] FLAG: --cores-total="0:320000"
I0603 12:59:25.224345 1 flags.go:52] FLAG: --daemonset-eviction-for-empty-nodes="false"
I0603 12:59:25.224350 1 flags.go:52] FLAG: --daemonset-eviction-for-occupied-nodes="true"
I0603 12:59:25.224355 1 flags.go:52] FLAG: --emit-per-nodegroup-metrics="false"
I0603 12:59:25.224360 1 flags.go:52] FLAG: --estimator="binpacking"
I0603 12:59:25.224364 1 flags.go:52] FLAG: --expander="least-waste"
I0603 12:59:25.224369 1 flags.go:52] FLAG: --expendable-pods-priority-cutoff="-10"
I0603 12:59:25.224374 1 flags.go:52] FLAG: --gce-concurrent-refreshes="1"
I0603 12:59:25.224379 1 flags.go:52] FLAG: --gpu-total="[]"
I0603 12:59:25.224383 1 flags.go:52] FLAG: --ignore-daemonsets-utilization="false"
I0603 12:59:25.224387 1 flags.go:52] FLAG: --ignore-mirror-pods-utilization="false"
I0603 12:59:25.224392 1 flags.go:52] FLAG: --ignore-taint="[]"
I0603 12:59:25.224396 1 flags.go:52] FLAG: --kubeconfig=""
I0603 12:59:25.224400 1 flags.go:52] FLAG: --kubernetes=""
I0603 12:59:25.224405 1 flags.go:52] FLAG: --leader-elect="true"
I0603 12:59:25.224415 1 flags.go:52] FLAG: --leader-elect-lease-duration="15s"
I0603 12:59:25.224421 1 flags.go:52] FLAG: --leader-elect-renew-deadline="10s"
I0603 12:59:25.224425 1 flags.go:52] FLAG: --leader-elect-resource-lock="leases"
I0603 12:59:25.224431 1 flags.go:52] FLAG: --leader-elect-resource-name="cluster-autoscaler"
I0603 12:59:25.224436 1 flags.go:52] FLAG: --leader-elect-resource-namespace=""
I0603 12:59:25.224440 1 flags.go:52] FLAG: --leader-elect-retry-period="2s"
I0603 12:59:25.224445 1 flags.go:52] FLAG: --log-backtrace-at=":0"
I0603 12:59:25.224453 1 flags.go:52] FLAG: --log-dir=""
I0603 12:59:25.224458 1 flags.go:52] FLAG: --log-file=""
I0603 12:59:25.224462 1 flags.go:52] FLAG: --log-file-max-size="1800"
I0603 12:59:25.224467 1 flags.go:52] FLAG: --logtostderr="true"
I0603 12:59:25.224471 1 flags.go:52] FLAG: --max-autoprovisioned-node-group-count="15"
I0603 12:59:25.224475 1 flags.go:52] FLAG: --max-bulk-soft-taint-count="10"
I0603 12:59:25.224479 1 flags.go:52] FLAG: --max-bulk-soft-taint-time="3s"
I0603 12:59:25.224484 1 flags.go:52] FLAG: --max-empty-bulk-delete="10"
I0603 12:59:25.224491 1 flags.go:52] FLAG: --max-failing-time="15m0s"
I0603 12:59:25.224495 1 flags.go:52] FLAG: --max-graceful-termination-sec="600"
I0603 12:59:25.224500 1 flags.go:52] FLAG: --max-inactivity="10m0s"
I0603 12:59:25.224504 1 flags.go:52] FLAG: --max-node-provision-time="15m0s"
I0603 12:59:25.224508 1 flags.go:52] FLAG: --max-nodes-total="0"
I0603 12:59:25.224512 1 flags.go:52] FLAG: --max-total-unready-percentage="45"
I0603 12:59:25.224517 1 flags.go:52] FLAG: --memory-total="0:6400000"
I0603 12:59:25.224522 1 flags.go:52] FLAG: --min-replica-count="0"
I0603 12:59:25.224527 1 flags.go:52] FLAG: --namespace="default"
I0603 12:59:25.224531 1 flags.go:52] FLAG: --new-pod-scale-up-delay="0s"
I0603 12:59:25.224535 1 flags.go:52] FLAG: --node-autoprovisioning-enabled="false"
I0603 12:59:25.224540 1 flags.go:52] FLAG: --node-deletion-delay-timeout="2m0s"
I0603 12:59:25.224544 1 flags.go:52] FLAG: --node-group-auto-discovery="[asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/eks-cluster]"
I0603 12:59:25.224554 1 flags.go:52] FLAG: --nodes="[]"
I0603 12:59:25.224558 1 flags.go:52] FLAG: --ok-total-unready-count="3"
I0603 12:59:25.224563 1 flags.go:52] FLAG: --one-output="false"
I0603 12:59:25.224567 1 flags.go:52] FLAG: --profiling="false"
I0603 12:59:25.224571 1 flags.go:52] FLAG: --regional="false"
I0603 12:59:25.224575 1 flags.go:52] FLAG: --scale-down-candidates-pool-min-count="50"
I0603 12:59:25.224579 1 flags.go:52] FLAG: --scale-down-candidates-pool-ratio="0.1"
I0603 12:59:25.224584 1 flags.go:52] FLAG: --scale-down-delay-after-add="10m0s"
I0603 12:59:25.224589 1 flags.go:52] FLAG: --scale-down-delay-after-delete="0s"
I0603 12:59:25.224593 1 flags.go:52] FLAG: --scale-down-delay-after-failure="3m0s"
I0603 12:59:25.224597 1 flags.go:52] FLAG: --scale-down-enabled="true"
I0603 12:59:25.224601 1 flags.go:52] FLAG: --scale-down-gpu-utilization-threshold="0.5"
I0603 12:59:25.224606 1 flags.go:52] FLAG: --scale-down-non-empty-candidates-count="30"
I0603 12:59:25.224610 1 flags.go:52] FLAG: --scale-down-unneeded-time="10m0s"
I0603 12:59:25.224615 1 flags.go:52] FLAG: --scale-down-unready-time="20m0s"
I0603 12:59:25.224619 1 flags.go:52] FLAG: --scale-down-utilization-threshold="0.5"
I0603 12:59:25.224624 1 flags.go:52] FLAG: --scale-up-from-zero="true"
I0603 12:59:25.224628 1 flags.go:52] FLAG: --scan-interval="10s"
I0603 12:59:25.224632 1 flags.go:52] FLAG: --skip-headers="false"
I0603 12:59:25.224636 1 flags.go:52] FLAG: --skip-log-headers="false"
I0603 12:59:25.224640 1 flags.go:52] FLAG: --skip-nodes-with-local-storage="false"
I0603 12:59:25.224645 1 flags.go:52] FLAG: --skip-nodes-with-system-pods="true"
I0603 12:59:25.224649 1 flags.go:52] FLAG: --status-config-map-name="cluster-autoscaler-status"
I0603 12:59:25.224654 1 flags.go:52] FLAG: --stderrthreshold="0"
I0603 12:59:25.224658 1 flags.go:52] FLAG: --unremovable-node-recheck-timeout="5m0s"
I0603 12:59:25.224662 1 flags.go:52] FLAG: --user-agent="cluster-autoscaler"
I0603 12:59:25.224668 1 flags.go:52] FLAG: --v="4"
I0603 12:59:25.224673 1 flags.go:52] FLAG: --vmodule=""
I0603 12:59:25.224677 1 flags.go:52] FLAG: --write-status-configmap="true"
I0603 12:59:25.224689 1 main.go:397] Cluster Autoscaler 1.22.0
I0603 12:59:25.254052 1 leaderelection.go:248] attempting to acquire leader lease default/cluster-autoscaler...
I0603 12:59:25.260116 1 leaderelection.go:352] lock is held by my-release-aws-cluster-autoscaler-59bcc57d8d-wdsx2 and has not yet expired
I0603 12:59:25.260289 1 leaderelection.go:253] failed to acquire lease default/cluster-autoscaler
I0603 12:59:28.722598 1 leaderelection.go:352] lock is held by my-release-aws-cluster-autoscaler-59bcc57d8d-wdsx2 and has not yet expired
I0603 12:59:28.722614 1 leaderelection.go:253] failed to acquire lease default/cluster-autoscaler
I0603 12:59:32.994729 1 leaderelection.go:352] lock is held by my-release-aws-cluster-autoscaler-59bcc57d8d-wdsx2 and has not yet expired
I0603 12:59:32.994747 1 leaderelection.go:253] failed to acquire lease default/cluster-autoscaler
I0603 12:59:36.603814 1 leaderelection.go:352] lock is held by my-release-aws-cluster-autoscaler-59bcc57d8d-wdsx2 and has not yet expired
I0603 12:59:36.603833 1 leaderelection.go:253] failed to acquire lease default/cluster-autoscaler
I0603 12:59:39.667698 1 leaderelection.go:352] lock is held by my-release-aws-cluster-autoscaler-59bcc57d8d-wdsx2 and has not yet expired
I0603 12:59:39.667717 1 leaderelection.go:253] failed to acquire lease default/cluster-autoscaler
I0603 12:59:42.715807 1 leaderelection.go:258] successfully acquired lease default/cluster-autoscaler
I0603 12:59:42.716107 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"Lease", Namespace:"default", Name:"cluster-autoscaler", UID:"93f0c17a-b6f5-4b46-998e-b3a46c38978e", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"49210", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' my-release-aws-cluster-autoscaler-c4fb847fb-r9wm9 became leader
I0603 12:59:42.719930 1 reflector.go:219] Starting reflector *v1.Pod (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:188
I0603 12:59:42.720069 1 reflector.go:255] Listing and watching *v1.Pod from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:188
I0603 12:59:42.720531 1 reflector.go:219] Starting reflector *v1.Pod (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:212
I0603 12:59:42.720652 1 reflector.go:255] Listing and watching *v1.Pod from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:212
I0603 12:59:42.720944 1 reflector.go:219] Starting reflector *v1.Node (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:246
I0603 12:59:42.721053 1 reflector.go:255] Listing and watching *v1.Node from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:246
I0603 12:59:42.721328 1 reflector.go:219] Starting reflector *v1.Node (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:246
I0603 12:59:42.721440 1 reflector.go:255] Listing and watching *v1.Node from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:246
I0603 12:59:42.721748 1 reflector.go:219] Starting reflector *v1beta1.PodDisruptionBudget (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309
I0603 12:59:42.721866 1 reflector.go:255] Listing and watching *v1beta1.PodDisruptionBudget from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309
I0603 12:59:42.722207 1 reflector.go:219] Starting reflector *v1.DaemonSet (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:320
I0603 12:59:42.722335 1 reflector.go:255] Listing and watching *v1.DaemonSet from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:320
I0603 12:59:42.722684 1 reflector.go:219] Starting reflector *v1.ReplicationController (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:329
I0603 12:59:42.722884 1 reflector.go:255] Listing and watching *v1.ReplicationController from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:329
I0603 12:59:42.723214 1 reflector.go:219] Starting reflector *v1.Job (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:338
I0603 12:59:42.723312 1 reflector.go:255] Listing and watching *v1.Job from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:338
I0603 12:59:42.723619 1 reflector.go:219] Starting reflector *v1.ReplicaSet (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:347
I0603 12:59:42.723715 1 reflector.go:255] Listing and watching *v1.ReplicaSet from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:347
I0603 12:59:42.723985 1 reflector.go:219] Starting reflector *v1.StatefulSet (1h0m0s) from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:356
I0603 12:59:42.724148 1 reflector.go:255] Listing and watching *v1.StatefulSet from k8s.io/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:356
W0603 12:59:42.828315 1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0603 12:59:43.229708 1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
I0603 12:59:43.230626 1 cloud_provider_builder.go:29] Building aws cloud provider.
I0603 12:59:43.230757 1 aws_util.go:81] fetching https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonEC2/current/us-east-1/index.json
I0603 12:59:43.232234 1 reflector.go:219] Starting reflector *v1.ReplicationController (0s) from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.232709 1 reflector.go:255] Listing and watching *v1.ReplicationController from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.233170 1 reflector.go:219] Starting reflector *v1.Namespace (0s) from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.233272 1 reflector.go:255] Listing and watching *v1.Namespace from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.233686 1 reflector.go:219] Starting reflector *v1.PodDisruptionBudget (0s) from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.233786 1 reflector.go:255] Listing and watching *v1.PodDisruptionBudget from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.234244 1 reflector.go:219] Starting reflector *v1.Node (0s) from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.234346 1 reflector.go:255] Listing and watching *v1.Node from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.234772 1 reflector.go:219] Starting reflector *v1beta1.CSIStorageCapacity (0s) from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.234874 1 reflector.go:255] Listing and watching *v1beta1.CSIStorageCapacity from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.235344 1 reflector.go:219] Starting reflector *v1.Service (0s) from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.235448 1 reflector.go:255] Listing and watching *v1.Service from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.235849 1 reflector.go:219] Starting reflector *v1.StatefulSet (0s) from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.323455 1 reflector.go:255] Listing and watching *v1.StatefulSet from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.323857 1 reflector.go:219] Starting reflector *v1.PersistentVolume (0s) from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.323887 1 reflector.go:255] Listing and watching *v1.PersistentVolume from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.324175 1 reflector.go:219] Starting reflector *v1.PersistentVolumeClaim (0s) from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.324184 1 reflector.go:255] Listing and watching *v1.PersistentVolumeClaim from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.324452 1 reflector.go:219] Starting reflector *v1.StorageClass (0s) from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.324462 1 reflector.go:255] Listing and watching *v1.StorageClass from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.324789 1 reflector.go:219] Starting reflector *v1.CSIDriver (0s) from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.324799 1 reflector.go:255] Listing and watching *v1.CSIDriver from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.325067 1 reflector.go:219] Starting reflector *v1.ReplicaSet (0s) from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.325076 1 reflector.go:255] Listing and watching *v1.ReplicaSet from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.325330 1 reflector.go:219] Starting reflector *v1.CSINode (0s) from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.325339 1 reflector.go:255] Listing and watching *v1.CSINode from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.325592 1 reflector.go:219] Starting reflector *v1.Pod (0s) from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.325600 1 reflector.go:255] Listing and watching *v1.Pod from k8s.io/client-go/informers/factory.go:134
I0603 12:59:43.523491 1 request.go:597] Waited for 197.81208ms due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/api/v1/pods?limit=500&resourceVersion=0
I0603 13:00:22.424110 1 aws_cloud_provider.go:374] Using static instance type i3p.16xlarge
I0603 13:00:22.424164 1 aws_cloud_provider.go:384] Successfully load 619 EC2 Instance Types [r5d.24xlarge x2iezn.metal m6a.metal m5d.metal g5.2xlarge i4i.16xlarge c6i.2xlarge c5ad.24xlarge m5d.2xlarge r5d.4xlarge m6gd.medium r5d.2xlarge m6a.24xlarge r6g.12xlarge p3 d3en.8xlarge m5n.xlarge c6a.xlarge c6id.12xlarge m4 m6id.2xlarge u-18tb1 c7g.12xlarge t4g.small g4dn.8xlarge dl1 m5n r5b m5zn.2xlarge c5n.9xlarge r3.4xlarge c6id.4xlarge r4.xlarge c6a.16xlarge r5.large f1 i3 r6i.large m5n.large c6a.24xlarge x2iedn.2xlarge c6gn.large x2gd.large u-9tb1.metal m6a.16xlarge x2iedn.xlarge m5d.16xlarge r5ad.2xlarge x2gd.metal inf1 h1 x2iezn.2xlarge c6i.12xlarge r6i.16xlarge r5n.2xlarge r5d r4.16xlarge m5.8xlarge d2.8xlarge u-3tb1.56xlarge m6g.2xlarge m6i r5dn.16xlarge c5d.4xlarge r5b.large r5n.16xlarge r5dn m5zn m5ad.2xlarge c5d m5 g3s.xlarge t3 c6gd.2xlarge c7g.xlarge vt1.24xlarge r6g.xlarge r5n.12xlarge m5n.metal x1.32xlarge m1.xlarge c7g.2xlarge c7g.8xlarge i3.2xlarge r6i.xlarge m5dn.4xlarge x1e.16xlarge g4dn.2xlarge z1d.metal c5.4xlarge m5.2xlarge r5n.24xlarge p2.16xlarge i3en.6xlarge g5g.4xlarge is4gen.large d2 m5ad.8xlarge g5.xlarge x2iedn m5dn.16xlarge r3.8xlarge c3.4xlarge m6i.16xlarge c5d.metal x2gd m5dn.8xlarge x2gd.8xlarge c5ad.2xlarge m5dn im4gn c5n.xlarge m6gd.12xlarge c6gd.large t3.micro g2 m1.small m6g.medium r6gd m6i.2xlarge c6a.large c5.metal x2iedn.24xlarge c5.24xlarge t2.micro m6gd.4xlarge im4gn.2xlarge m6a.32xlarge d3.xlarge c5a.16xlarge c5ad.8xlarge r5d.metal c6gn.2xlarge t3.nano r6g.2xlarge p4de.24xlarge c5.9xlarge r5b.12xlarge p3dn.24xlarge r5.12xlarge m3.xlarge c5.18xlarge c6gd.metal t4g.medium c6gd.medium r6i.12xlarge r4 c7g.large x2gd.16xlarge x2idn.metal d3en.2xlarge r4.8xlarge r6i.4xlarge c6id.32xlarge inf1.6xlarge g4dn.metal r5d.16xlarge r5ad.xlarge i4i.32xlarge c6gd.16xlarge r5dn.metal m5n.24xlarge x2iedn.metal r5ad.24xlarge i3en.2xlarge is4gen.8xlarge r5b.metal m5n.4xlarge x2iedn.16xlarge d3en.12xlarge i3en.large g5.12xlarge t3a.xlarge m5zn.3xlarge r5b.2xlarge f1.2xlarge r5a.2xlarge c6i.32xlarge p3.16xlarge u-12tb1.metal m5dn.metal m5a.16xlarge g3.8xlarge i2 u-6tb1.metal inf1.24xlarge d3en.4xlarge x2iezn.8xlarge m5a.4xlarge c3.2xlarge inf1.2xlarge c4.2xlarge d3en.xlarge r5a.xlarge x1 u-6tb1.112xlarge x2iezn.12xlarge m6a.xlarge r5.2xlarge d3.4xlarge m5dn.12xlarge r5dn.24xlarge m2.2xlarge c6g.2xlarge t3.2xlarge f1.16xlarge m6a.large m5d.12xlarge vt1.6xlarge m5a.12xlarge g4ad.4xlarge m5n.8xlarge c6i.metal d2.xlarge m4.10xlarge m5.12xlarge g4ad.16xlarge r5dn.2xlarge t2.small mac1.metal c5a.24xlarge m5d.8xlarge r6g.4xlarge c5a.8xlarge d3en.6xlarge m6i.xlarge a1.2xlarge c1.xlarge m5zn.large x2gd.12xlarge r5dn.12xlarge c6id.2xlarge r5dn.4xlarge p2.8xlarge c7g.16xlarge r5n.metal c6g.large c6g.4xlarge r5n m5dn.24xlarge z1d.3xlarge u-6tb1.56xlarge c7g.medium g5.4xlarge c6gn c4 c5d.2xlarge r5ad.12xlarge m3.2xlarge c3.xlarge m6gd.xlarge u-24tb1 c6id c5ad.4xlarge r5n.xlarge i3en.xlarge r5ad.4xlarge m6a.8xlarge m5.16xlarge t3.medium c6gn.8xlarge c6id.16xlarge m5dn.2xlarge c6i.xlarge m4.16xlarge t3.large g5g.metal z1d d3.2xlarge c6gn.12xlarge r5ad.16xlarge m3 m6i.8xlarge r6g.metal c6id.metal c6gn.16xlarge m6i.4xlarge p3dn i3en.12xlarge m5ad.24xlarge t3a.large m5dn.large i3.large m6a x2idn.16xlarge x2idn.32xlarge g4ad.xlarge m6id.4xlarge r5a.24xlarge m5n.12xlarge r3.large c6a.32xlarge m5d.large m6a.48xlarge c6a.metal m5d.24xlarge c5d.18xlarge p4d x2idn mac2.metal i2.xlarge a1.4xlarge m5.large t2.2xlarge u-18tb1.metal r5dn.8xlarge x1e.8xlarge g4dn.4xlarge r5d.large m4.xlarge t4g.large r6g.large a1 c1.medium m6id.12xlarge g5.24xlarge m5d.4xlarge im4gn.8xlarge c5ad.12xlarge r5n.large g2.8xlarge m6g.large x2iezn.4xlarge m5zn.xlarge r3.xlarge c6i.16xlarge c6a.8xlarge i4i c5n m5a.24xlarge r5dn.xlarge m5a.8xlarge g4ad.2xlarge r4.large r5b.xlarge t3a.nano c5.xlarge im4gn.xlarge r6gd.xlarge c5 c5a.12xlarge m6i.metal i3en.3xlarge c5.2xlarge r5.metal t2.xlarge t3a.medium im4gn.16xlarge r6g m5n.2xlarge c5a.xlarge c5a.large c6g.metal h1.8xlarge r5ad.8xlarge h1.2xlarge m5.24xlarge im4gn.4xlarge m5ad.xlarge c6gd r5.24xlarge c4.large c5d.24xlarge r6i r5d.8xlarge c5n.metal r6gd.8xlarge c4.4xlarge m2.xlarge c6id.xlarge r5.xlarge i3.8xlarge z1d.12xlarge c6gd.xlarge z1d.6xlarge p3.2xlarge r6gd.large m6a.12xlarge x2gd.xlarge m3.large t2.large c5a.4xlarge dl1.24xlarge r6g.8xlarge i3.xlarge g4dn.16xlarge m4.large r5b.24xlarge m6g.16xlarge d3.8xlarge c6g.medium u-12tb1 is4gen.xlarge g5g m6g.4xlarge t4g.nano c6a g3.16xlarge m6gd.8xlarge g4dn t4g.micro g3 is4gen.medium h1.4xlarge r5a.8xlarge c6g.12xlarge m6gd.2xlarge g4ad r6i.metal m6gd.large m5dn.xlarge m5a.large m6id.xlarge inf1.xlarge a1.metal c6a.4xlarge c6gd.4xlarge m5zn.metal a1.xlarge r5b.8xlarge im4gn.large i3en h1.16xlarge m3.medium c6g.xlarge t3.small c6a.48xlarge c6id.24xlarge g4dn.xlarge r5a.large i3p.16xlarge r5d.12xlarge c6g.8xlarge c5d.xlarge r6g.medium t3.xlarge a1.medium g2.2xlarge r6gd.12xlarge c5ad.large c6i.8xlarge m6gd.metal t2.nano m5d m6id c5.12xlarge r5n.8xlarge c3 m6g.metal c5d.9xlarge c6g.16xlarge c6gn.metal mac2 x1.16xlarge g3.4xlarge r6gd.metal m5n.16xlarge m4.4xlarge c6id.large m6g.8xlarge m6gd x2gd.medium f1.4xlarge c5ad.xlarge i2.2xlarge r5dn.large g5.16xlarge r5.16xlarge m6i.large is4gen.2xlarge r5.4xlarge g4dn.12xlarge c4.xlarge r4.2xlarge m4.2xlarge c7g.4xlarge m6i.24xlarge r6gd.16xlarge c5a.2xlarge p3.8xlarge c6a.12xlarge mac1 m5.xlarge z1d.large p4d.24xlarge r6gd.4xlarge m5zn.12xlarge m6a.4xlarge m6i.32xlarge i4i.8xlarge m6id.32xlarge m5.metal m6gd.16xlarge c7g r3 m5zn.6xlarge x2gd.4xlarge c5d.12xlarge g5.8xlarge g5g.8xlarge m5a.xlarge c6i.large c6gn.4xlarge c4.8xlarge u-24tb1.metal t3a.small m6id.large r5n.4xlarge g4ad.8xlarge i4i.2xlarge p4de c3.8xlarge c5n.4xlarge i3.metal i3en.24xlarge m1.medium m5ad.large r5d.xlarge r6i.8xlarge p2 m5ad.4xlarge x2iedn.4xlarge c3.large m2.4xlarge c5n.18xlarge t2.medium r6gd.2xlarge x2iezn u-12tb1.112xlarge t3a.2xlarge d2.2xlarge c6i u-6tb1 c6i.24xlarge i4i.large r5a.16xlarge x2iedn.8xlarge g5 x1e.2xlarge i2.8xlarge r6i.2xlarge r5ad.large r5 d2.4xlarge m5ad.16xlarge m5ad.12xlarge m6id.8xlarge z1d.2xlarge c6id.8xlarge r6g.16xlarge i2.4xlarge c5ad.16xlarge c6a.2xlarge r5.8xlarge x1e.32xlarge u-9tb1 i2.large m5a.2xlarge g5g.16xlarge g5.48xlarge r5b.16xlarge hs1.8xlarge c5n.2xlarge i4i.4xlarge t4g.2xlarge r6gd.medium z1d.xlarge vt1.3xlarge c6gd.12xlarge c6gd.8xlarge m6id.24xlarge m6a.2xlarge r4.4xlarge i3.4xlarge cc2.8xlarge m6id.metal r5a.4xlarge m6g x1e r5b.4xlarge r3.2xlarge m6g.12xlarge c6gn.xlarge is4gen.4xlarge c5d.large m6g.xlarge t1.micro m5.4xlarge r6i.24xlarge c6gn.medium x1e.xlarge i4i.xlarge r5a.12xlarge x2iedn.32xlarge i4i.metal r6i.32xlarge c6g x2gd.2xlarge g5g.xlarge m6i.12xlarge m5d.xlarge t3a.micro x1e.4xlarge x2iezn.6xlarge u-9tb1.112xlarge g5g.2xlarge c5.large i3en.metal p2.xlarge i3.16xlarge c5n.large t4g.xlarge m6id.16xlarge c6i.4xlarge x2idn.24xlarge a1.large cr1.8xlarge m1.large vt1]
E0603 13:00:22.664909 1 aws_manager.go:264] Failed to regenerate ASG cache: cannot autodiscover ASGs: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
status code: 403, request id: 00180f24-0fce-4112-a17a-e92d628231b3
F0603 13:00:22.664929 1 aws_cloud_provider.go:389] Failed to create AWS Manager: cannot autodiscover ASGs: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
status code: 403, request id: 00180f24-0fce-4112-a17a-e92d628231b3
goroutine 72 [running]:
k8s.io/klog/v2.stacks(0xc00000e001, 0xc0008d6a80, 0x135, 0x17b)
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/klog/v2/klog.go:1026 +0xb9
k8s.io/klog/v2.(*loggingT).output(0x61261c0, 0xc000000003, 0x0, 0x0, 0xc013db0af0, 0x0, 0x50a7f44, 0x15, 0x185, 0x0)
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/klog/v2/klog.go:975 +0x1e5
k8s.io/klog/v2.(*loggingT).printf(0x61261c0, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x41187cf, 0x20, 0xc013dcab50, 0x1, ...)
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/klog/v2/klog.go:753 +0x19a
k8s.io/klog/v2.Fatalf(...)
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/klog/v2/klog.go:1514
k8s.io/autoscaler/cluster-autoscaler/cloudprovider/aws.BuildAWS(0x3fe0000000000000, 0x3fe0000000000000, 0x8bb2c97000, 0x1176592e000, 0xa, 0x0, 0x4e200, 0x0, 0x186a0000000000, 0x0, ...)
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/cloudprovider/aws/aws_cloud_provider.go:389 +0x966
k8s.io/autoscaler/cluster-autoscaler/cloudprovider/builder.buildCloudProvider(0x3fe0000000000000, 0x3fe0000000000000, 0x8bb2c97000, 0x1176592e000, 0xa, 0x0, 0x4e200, 0x0, 0x186a0000000000, 0x0, ...)
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/cloudprovider/builder/builder_all.go:73 +0x1b8
k8s.io/autoscaler/cluster-autoscaler/cloudprovider/builder.NewCloudProvider(0x3fe0000000000000, 0x3fe0000000000000, 0x8bb2c97000, 0x1176592e000, 0xa, 0x0, 0x4e200, 0x0, 0x186a0000000000, 0x0, ...)
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/cloudprovider/builder/cloud_provider_builder.go:45 +0x230
k8s.io/autoscaler/cluster-autoscaler/core.initializeDefaultOptions(0xc0009576e8, 0xc00045b4a0, 0x451085)
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/core/autoscaler.go:101 +0x318
k8s.io/autoscaler/cluster-autoscaler/core.NewAutoscaler(0x3fe0000000000000, 0x3fe0000000000000, 0x8bb2c97000, 0x1176592e000, 0xa, 0x0, 0x4e200, 0x0, 0x186a0000000000, 0x0, ...)
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/core/autoscaler.go:65 +0x45
main.buildAutoscaler(0xc000067700, 0x46e450, 0xc0001cd080, 0x7ff3225644e0)
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/main.go:343 +0x3b8
main.run(0xc0003545f0)
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/main.go:349 +0x4a
main.main.func2(0x469a390, 0xc0006132c0)
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/main.go:454 +0x2a
created by k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run
/gopath/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:211 +0x11bThat's a completely different issue, please read the logs you've pasted:
E0603 13:00:22.664909 1 aws_manager.go:264] Failed to regenerate ASG cache: cannot autodiscover ASGs: WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
status code: 403, request id: 00180f24-0fce-4112-a17a-e92d628231b3I'm closing this issue for now as it's not a bug with the chart.
ashishjullia
commented
2 years ago @gjtempleton Thanks for all your help, I am able to solve the issue.
@roberthbailey @mgoodness @gjtempleton @sc250024
While trying to run the following terraform code using helm provider:
I'm getting the following error:
Also, "autoDiscovery.clusterName" is always "null"