bnaydenov
commented
1 year ago @shapirov103 you can close this one
Closed bnaydenov closed 1 year ago
bnaydenov
commented
1 year ago @shapirov103 you can close this one
softmates
commented
1 year ago @bnaydenov how is it working on 1.22 without the policy "ec2:DescribeInstanceTypes" ?
bnaydenov
commented
1 year ago @softmates most most likely is due to different version of the helm chart which autoscaler uses for different version of eks kubernetes. Check this file: https://github.com/aws-quickstart/cdk-eks-blueprints/blob/c03512bfd71ab735868da7c6acb43cfbb0e51dfe/lib/addons/cluster-autoscaler/index.ts#L44
eks k8s 1.21 and 1.22 are using the same helm chart version, but in 1.23 is different
const versionMap = new Map([
[KubernetesVersion.V1_23, "9.21.0"],
[KubernetesVersion.V1_22, "9.13.1"],
[KubernetesVersion.V1_21, "9.13.1"],
[KubernetesVersion.V1_20, "9.9.2"],
[KubernetesVersion.V1_19, "9.4.0"],
[KubernetesVersion.V1_18, "9.4.0"],
]);Yep, it make sense. Looking at the spec don't see a need for additional policy "ec2:DescribeInstanceTypes" refer the comparison
https://artifacthub.io/packages/helm/cluster-autoscaler/cluster-autoscaler/9.13.0
spec: additionalPolicies: node: | [ {"Effect":"Allow","Action":["autoscaling:DescribeAutoScalingGroups","autoscaling:DescribeAutoScalingInstances","autoscaling:DescribeLaunchConfigurations","autoscaling:DescribeTags","autoscaling:SetDesiredCapacity","autoscaling:TerminateInstanceInAutoScalingGroup"],"Resource":"*"} ] ...
https://artifacthub.io/packages/helm/cluster-autoscaler/cluster-autoscaler/9.21.0
bnaydenov
commented
1 year ago @softmates i was inspired from this issue:
https://github.com/kubernetes/autoscaler/issues/3216 more specifically this comment https://github.com/kubernetes/autoscaler/issues/3216#issuecomment-1047164006
bnaydenov
commented
1 year ago this changes are released in 1.4.1 so I will close the issue
Describe the bug
When cdk blueprint version
1.4is used andClusterAutoScalerAddOnis installed oneks kubernetes 1.23pod forblueprints-addon-cluster-autoscaler-aws-cluster-autoscaleris fails to start.The same setup works without problems on
eks kubernetes 1.21and1.22Expected Behavior
When installing
ClusterAutoScalerAddOnoneks kubernetes 1.23pod forblueprints-addon-cluster-autoscaler-aws-cluster-autoscalersuppose to start without errors.Current Behavior
pod
blueprints-addon-cluster-autoscaler-aws-cluster-autoscalerstarts and during starting phase which takes about 15-20 sec pod crash with following errors:The main error is
Failed to generate AWS EC2 Instance Types: UnauthorizedOperation: You are not authorized to perform this operation.Reproduction Steps
just use
cdk blueprint 1.4to spin up brand neweks k8s cluster 1.23and usingClusterAutoScalerAddOncdk deploystep will be successful, but after that podblueprints-addon-cluster-autoscaler-aws-cluster-autoscalerwill crash and can not be started.Possible Solution
I have found what is wrong and will prepare PR to fix this.
TLDR: We need to add missing policy
ec2:DescribeInstanceTypesfor cluster-autoscaler IAM statements here:https://github.com/aws-quickstart/cdk-eks-blueprints/blob/c03512bfd71ab735868da7c6acb43cfbb0e51dfe/lib/addons/cluster-autoscaler/index.ts#L79
For more info check here: https://github.com/kubernetes/autoscaler/issues/3216 https://github.com/particuleio/terraform-kubernetes-addons/pull/1320
I have created local monkey patch of
addon/cluster-autoscalerin https://github.com/aws-quickstart/cdk-eks-blueprints/blob/c03512bfd71ab735868da7c6acb43cfbb0e51dfe/lib/addons/cluster-autoscaler/index.ts#L79 with adding this missing policy"ec2:DescribeInstanceTypes"and everything is working as expected.Additional Information/Context
No response
CDK CLI Version
2.50.0 (build 4c11af6)
EKS Blueprints Version
1.4.0
Node.js Version
v16.17.0
Environment details (OS name and version, etc.)
Mac OS Monterey - Version 12.6
Other information
No response