Closed s-tokutake closed 4 years ago
Hit this error today.
I added the following to my IAM Policy to successfully create the required role:
{
"Sid": "VisualEditor6",
"Effect": "Allow",
"Action": [
"iam:CreateInstanceProfile",
"iam:DeleteInstanceProfile",
"iam:GetRole",
"iam:GetInstanceProfile",
"iam:RemoveRoleFromInstanceProfile",
"iam:CreateRole",
"iam:DeleteRole",
"iam:AttachRolePolicy",
"iam:PutRolePolicy",
"iam:ListInstanceProfiles",
"iam:AddRoleToInstanceProfile",
"iam:ListInstanceProfilesForRole",
"iam:PassRole",
"iam:CreateServiceLinkedRole",
"iam:DetachRolePolicy",
"iam:DeleteRolePolicy",
"iam:DeleteServiceLinkedRole",
"iam:GetRolePolicy"
],
"Resource": [
"arn:aws:iam::*:role/eksctl-*",
"arn:aws:iam::*:instance-profile/eksctl-*"
]
}
Hit this error today.
I added the following to my IAM Policy to successfully create the required role:
{ "Sid": "VisualEditor6", "Effect": "Allow", "Action": [ "iam:CreateInstanceProfile", "iam:DeleteInstanceProfile", "iam:GetRole", "iam:GetInstanceProfile", "iam:RemoveRoleFromInstanceProfile", "iam:CreateRole", "iam:DeleteRole", "iam:AttachRolePolicy", "iam:PutRolePolicy", "iam:ListInstanceProfiles", "iam:AddRoleToInstanceProfile", "iam:ListInstanceProfilesForRole", "iam:PassRole", "iam:CreateServiceLinkedRole", "iam:DetachRolePolicy", "iam:DeleteRolePolicy", "iam:DeleteServiceLinkedRole", "iam:GetRolePolicy" ], "Resource": [ "arn:aws:iam::*:role/eksctl-*", "arn:aws:iam::*:instance-profile/eksctl-*" ] }
I have these permissions in place but still got the error.
Same thing over here, same permissions and still have the error.
I also tried with these suggested IAM permissions from another thread, same results as yours.
https://github.com/weaveworks/eksctl/issues/204#issuecomment-628435072 https://github.com/weaveworks/eksctl/issues/204#issuecomment-631630355
I just tried right now temporally setting the AWS Managed policy "AdministratorAccess" to the eksctl IAM user and everything worked as expected, so, maybe we can say this is permissions configuration issue :man_shrugging: point is... which ones are we missing...
The following policy allows me to deploy an EKS cluster using ec2 spot instances using eksctl version 0.19.0
IAM Polcy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor100",
"Effect": "Allow",
"Action": [
"ssm:PutParameter",
"ssm:DeleteParameter",
"ssm:GetParameterHistory",
"ssm:GetParametersByPath",
"ssm:GetParameters",
"ssm:GetParameter",
"ssm:DeleteParameters"
],
"Resource": "*"
},
{
"Sid": "VisualEditor101",
"Effect": "Allow",
"Action": "ssm:DescribeParameters",
"Resource": "*"
},
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"autoscaling:CreateLaunchConfiguration",
"autoscaling:DescribeScalingActivities",
"autoscaling:DeleteLaunchConfiguration",
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeLaunchConfigurations",
"autoscaling:UpdateAutoScalingGroup",
"autoscaling:DeleteAutoScalingGroup",
"autoscaling:CreateAutoScalingGroup"
],
"Resource": "*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "cloudformation:*",
"Resource": "*"
},
{
"Sid": "VisualEditor2",
"Effect": "Allow",
"Action": "ec2:DeleteInternetGateway",
"Resource": "arn:aws:ec2:*:*:internet-gateway/*"
},
{
"Sid": "VisualEditor3",
"Effect": "Allow",
"Action": [
"ec2:AuthorizeSecurityGroupIngress",
"ec2:DeleteSubnet",
"ec2:DescribeAddresses",
"ec2:DeleteTags",
"ec2:CreateNatGateway",
"ec2:CreateVpc",
"ec2:AttachInternetGateway",
"ec2:DescribeVpcAttribute",
"ec2:DeleteRouteTable",
"ec2:AssociateRouteTable",
"ec2:DescribeInternetGateways",
"ec2:DescribeAvailabilityZones",
"ec2:CreateRoute",
"ec2:CreateInternetGateway",
"ec2:RevokeSecurityGroupEgress",
"ec2:CreateSecurityGroup",
"ec2:ModifyVpcAttribute",
"ec2:DeleteInternetGateway",
"ec2:DescribeKeyPairs",
"ec2:DescribeRouteTables",
"ec2:ReleaseAddress",
"ec2:AuthorizeSecurityGroupEgress",
"ec2:DeleteLaunchTemplate",
"ec2:ImportKeyPair",
"ec2:DescribeLaunchTemplates",
"ec2:DescribeTags",
"ec2:CreateTags",
"ec2:DeleteRoute",
"ec2:DescribeLaunchTemplateVersions",
"ec2:CreateRouteTable",
"ec2:RunInstances",
"ec2:DetachInternetGateway",
"ec2:DescribeNatGateways",
"ec2:DisassociateRouteTable",
"ec2:AllocateAddress",
"ec2:DescribeSecurityGroups",
"ec2:RevokeSecurityGroupIngress",
"ec2:DescribeImages",
"ec2:CreateLaunchTemplate",
"ec2:DescribeVpcs",
"ec2:DescribeImageAttribute",
"ec2:DeleteSecurityGroup",
"ec2:DeleteNatGateway",
"ec2:DeleteVpc",
"ec2:CreateSubnet",
"ec2:DescribeSubnets",
"ec2:ModifySubnetAttribute"
],
"Resource": "*"
},
{
"Sid": "VisualEditor4",
"Effect": "Allow",
"Action": [
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability",
"ecr:CompleteLayerUpload",
"ecr:DescribeImages",
"ecr:DescribeRepositories",
"ecr:GetDownloadUrlForLayer",
"ecr:InitiateLayerUpload",
"ecr:ListImages",
"ecr:PutImage",
"ecr:UploadLayerPart"
],
"Resource": "*"
},
{
"Sid": "VisualEditor41",
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken"
],
"Resource": "*"
},
{
"Sid": "VisualEditor5",
"Effect": "Allow",
"Action": "eks:*",
"Resource": "*"
},
{
"Sid": "VisualEditor6",
"Effect": "Allow",
"Action": [
"iam:CreateInstanceProfile",
"iam:DeleteInstanceProfile",
"iam:GetRole",
"iam:GetInstanceProfile",
"iam:RemoveRoleFromInstanceProfile",
"iam:CreateRole",
"iam:DeleteRole",
"iam:AttachRolePolicy",
"iam:PutRolePolicy",
"iam:ListInstanceProfiles",
"iam:AddRoleToInstanceProfile",
"iam:ListInstanceProfilesForRole",
"iam:PassRole",
"iam:CreateServiceLinkedRole",
"iam:DetachRolePolicy",
"iam:DeleteRolePolicy",
"iam:DeleteServiceLinkedRole",
"iam:GetRolePolicy"
],
"Resource": [
"arn:aws:iam::*:role/eksctl-*",
"arn:aws:iam::*:instance-profile/eksctl-*"
]
}
]
}
The following policy allows me to deploy an EKS cluster using ec2 spot instances using eksctl version 0.19.0
IAM Polcy
{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor100", "Effect": "Allow", "Action": [ "ssm:PutParameter", "ssm:DeleteParameter", "ssm:GetParameterHistory", "ssm:GetParametersByPath", "ssm:GetParameters", "ssm:GetParameter", "ssm:DeleteParameters" ], "Resource": "*" }, { "Sid": "VisualEditor101", "Effect": "Allow", "Action": "ssm:DescribeParameters", "Resource": "*" }, { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "autoscaling:CreateLaunchConfiguration", "autoscaling:DescribeScalingActivities", "autoscaling:DeleteLaunchConfiguration", "autoscaling:DescribeAutoScalingGroups", "autoscaling:DescribeLaunchConfigurations", "autoscaling:UpdateAutoScalingGroup", "autoscaling:DeleteAutoScalingGroup", "autoscaling:CreateAutoScalingGroup" ], "Resource": "*" }, { "Sid": "VisualEditor1", "Effect": "Allow", "Action": "cloudformation:*", "Resource": "*" }, { "Sid": "VisualEditor2", "Effect": "Allow", "Action": "ec2:DeleteInternetGateway", "Resource": "arn:aws:ec2:*:*:internet-gateway/*" }, { "Sid": "VisualEditor3", "Effect": "Allow", "Action": [ "ec2:AuthorizeSecurityGroupIngress", "ec2:DeleteSubnet", "ec2:DescribeAddresses", "ec2:DeleteTags", "ec2:CreateNatGateway", "ec2:CreateVpc", "ec2:AttachInternetGateway", "ec2:DescribeVpcAttribute", "ec2:DeleteRouteTable", "ec2:AssociateRouteTable", "ec2:DescribeInternetGateways", "ec2:DescribeAvailabilityZones", "ec2:CreateRoute", "ec2:CreateInternetGateway", "ec2:RevokeSecurityGroupEgress", "ec2:CreateSecurityGroup", "ec2:ModifyVpcAttribute", "ec2:DeleteInternetGateway", "ec2:DescribeKeyPairs", "ec2:DescribeRouteTables", "ec2:ReleaseAddress", "ec2:AuthorizeSecurityGroupEgress", "ec2:DeleteLaunchTemplate", "ec2:ImportKeyPair", "ec2:DescribeLaunchTemplates", "ec2:DescribeTags", "ec2:CreateTags", "ec2:DeleteRoute", "ec2:DescribeLaunchTemplateVersions", "ec2:CreateRouteTable", "ec2:RunInstances", "ec2:DetachInternetGateway", "ec2:DescribeNatGateways", "ec2:DisassociateRouteTable", "ec2:AllocateAddress", "ec2:DescribeSecurityGroups", "ec2:RevokeSecurityGroupIngress", "ec2:DescribeImages", "ec2:CreateLaunchTemplate", "ec2:DescribeVpcs", "ec2:DescribeImageAttribute", "ec2:DeleteSecurityGroup", "ec2:DeleteNatGateway", "ec2:DeleteVpc", "ec2:CreateSubnet", "ec2:DescribeSubnets", "ec2:ModifySubnetAttribute" ], "Resource": "*" }, { "Sid": "VisualEditor4", "Effect": "Allow", "Action": [ "ecr:BatchGetImage", "ecr:BatchCheckLayerAvailability", "ecr:CompleteLayerUpload", "ecr:DescribeImages", "ecr:DescribeRepositories", "ecr:GetDownloadUrlForLayer", "ecr:InitiateLayerUpload", "ecr:ListImages", "ecr:PutImage", "ecr:UploadLayerPart" ], "Resource": "*" }, { "Sid": "VisualEditor41", "Effect": "Allow", "Action": [ "ecr:GetAuthorizationToken" ], "Resource": "*" }, { "Sid": "VisualEditor5", "Effect": "Allow", "Action": "eks:*", "Resource": "*" }, { "Sid": "VisualEditor6", "Effect": "Allow", "Action": [ "iam:CreateInstanceProfile", "iam:DeleteInstanceProfile", "iam:GetRole", "iam:GetInstanceProfile", "iam:RemoveRoleFromInstanceProfile", "iam:CreateRole", "iam:DeleteRole", "iam:AttachRolePolicy", "iam:PutRolePolicy", "iam:ListInstanceProfiles", "iam:AddRoleToInstanceProfile", "iam:ListInstanceProfilesForRole", "iam:PassRole", "iam:CreateServiceLinkedRole", "iam:DetachRolePolicy", "iam:DeleteRolePolicy", "iam:DeleteServiceLinkedRole", "iam:GetRolePolicy" ], "Resource": [ "arn:aws:iam::*:role/eksctl-*", "arn:aws:iam::*:instance-profile/eksctl-*" ] } ] }
Thank youuuuu!!! hehehe, this set of permissions worked for me. I'm also using Spot instances and public/private endpoint with IP whitelisting.
Did you got them with trial and error or from some doc/place?
You are very welcome. 95% of the work was done by these fine folks https://github.com/weaveworks/eksctl/issues/204.
When you delete your cluster please double check the AWS Console and make sure the Cloudformation stacks which we created by eksctl are dropped cleanly.
I have been caught out in the past and been left with a bill I didn't expect! Cloudwatch billing events are essential as costs can run away with themselves.
Debugging these permissions was a case of watching Cloudformation Events , seeing the failures, understanding what was going on, updating my IAM Policy and going around the loop again.
I really wish eksctl.io would publish an IAM Policy on their site and this would have been a whole lot easier.
You are very welcome. 95% of the work was done by these fine folks #204.
When you delete your cluster please double check the AWS Console and make sure the Cloudformation stacks which we created by eksctl are dropped cleanly.
I have been caught out in the past and been left with a bill I didn't expect! Cloudwatch billing events are essential as costs can run away with themselves.
Debugging these permissions was a case of watching Cloudformation Events , seeing the failures, understanding what was going on, updating my IAM Policy and going around the loop again.
I really wish eksctl.io would publish an IAM Policy on their site and this would have been a whole lot easier.
Man that's a lot of work! (trial and error with EKS), which doesn't bootstrap as fast as a kops cluster. Thanks for that :wink:
I've been checking and I always got the Cloudformation stack correctly deleted, thanks for the reminder!
Hi, thanks for reporting this! It seems this is needed quite a bit. I will work on documenting the policies in the coming days (tracked via #204).
I've tried all of the IAM policies discussed here and in #204 and still get this error
Hi @ryanvade is it the exact same error what you are getting? Can you give us more details like logs and a redacted version of the config file you used?
eksctl Version: 0.24.0
eksctl create cluster --region us-east-1 --zones=us-east-1a,us-east-1b,us-east-1c --name=test
ends up with Role with arn: arn:aws:iam::xxxxxxxx:role/eksctl-test-cluster-ServiceRole-xxxxxxxx, could not be assumed because it does not exist or the trusted entity is not correct given the different policies mentioned in this and other threads.
Hi @ryanvade I can't reproduce this error with my accounts. Can you please run the same command with -v 4
and post all the logs?
In case anyone else encounters this, I got the same error as @ryanvade with eksctl 0.27.0 and noticed the following error in CloudTrail logs:
CreateServiceLinkedRole
AccessDenied
User: arn:aws:iam::xxx:user/MyUser is not authorized to perform: iam:CreateServiceLinkedRole on resource:
arn:aws:iam::xxx:role/aws-service-role/eks.amazonaws.com/AWSServiceRoleForAmazonEKS
It worked after adding the below permission to the "IamLimitedAccess" policy listed in the docs after substituting our account number for "xxx":
{
"Effect": "Allow",
"Action": "iam:CreateServiceLinkedRole",
"Resource": "arn:aws:iam::xxx:role/aws-service-role/eks*"
}
Redacted config file, in case it's helpful for reproducing:
kind: ClusterConfig
apiVersion: eksctl.io/v1alpha5
metadata:
name: my-cluster
region: us-east-1
version: "1.17"
tags:
MyTag: "Tag Value"
nodeGroups:
- name: persistent
instanceType: m5.large
desiredCapacity: 2
volumeSize: 80
preBootstrapCommands:
- echo "preBootstrap"
- name: compute-cpu
minSize: 0
maxSize: 5
volumeSize: 80
instancesDistribution:
instanceTypes: ["m5.large"]
onDemandBaseCapacity: 0
onDemandPercentageAboveBaseCapacity: 0
preBootstrapCommands:
- echo "preBootstrap"
- name: compute-gpu
minSize: 0
maxSize: 1
volumeSize: 100
instancesDistribution:
instanceTypes: ["g4dn.xlarge"]
onDemandBaseCapacity: 0
onDemandPercentageAboveBaseCapacity: 0
preBootstrapCommands:
- echo "preBootstrap"
git:
repo:
url: <my git repo>
branch: master
paths:
- base
fluxPath: flux/
user: gitops
email: <my email>
operator:
commitOperatorManifests: true
namespace: "flux"
withHelm: true
bootstrapProfile:
source: app-dev
revision: master
outputPath: base/
my EKS role was missing in
Trust relationships
{
"Effect": "Allow",
"Principal": {
"Service": "eks.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
It seems error response is bad.
could not be assumed because it does not exist or the trusted entity is not correct
should be
Unable to proceed, cannot describe custom KMS key
.
Releted to https://github.com/aws/containers-roadmap/issues/1533
The following KMS key's policy solved it for me.
{
"Sid": "Allow EKS cluster role to view the key during the updates",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::XXX:role/XXX"
},
"Action": "kms:DescribeKey",
"Resource": "*"
}
I found this issue because I create EKS cluster with Terraform AWS module.
I've extended the required permissions but iam:PassRole and iam:CreateServiceLinkedRole should not be used with *. Any idea how to prevent it and make them more secure?
JSON EKS Policies:
What happened?
Fail to create new cluster.
Error messge is
Role with arn: arn:aws:iam::xxxxxxxxxxxxx:role/eksctl-prd-cluster-ServiceRole-xxxxxxxxx, could not be assumed because it does not exist or the trusted entity is not correct (Service: AmazonEKS; Status Code: 400; Error Code: InvalidParameterException; Request ID: fc9b7780-6fb4-4620-8943-b523bxxxxxxx)
What you expected to happen?
To create cluster successfully.
How to reproduce it?
exec
eksctl create cluster -f cluster.yaml
cluster.yaml is below.
Anything else we need to know?
--verbose 5
), the ServiceRole is created successfully, but Create ControlPlane by using the ServiceRole fail.Versions
Logs