search

eksctl-io / eksctl

The official CLI for Amazon EKS
https://eksctl.io
Other
4.93k stars 1.41k forks source link

Improvements around setting up IAM (also for multiple clusters) #573

Closed StevenACoffman closed 3 years ago

StevenACoffman commented 5 years ago

Currently, when I launch a cluster using eksctl, I need to have the access rights to create IAM policies, profiles, and roles.

I believe it is possible to have an administrator create these beforehand and then allow users without IAM creation privileges to launch clusters using these pre-existing policies. It would be nice if this was better documented.

Our use case is that we have a pre-existing vpc, and we want developers to be able to spin up an ephemeral eks clusters inside it, but not be able to accidentally modify IAM permissions in a way that leaves us vulnerable.

Related: #508

StevenACoffman commented 5 years ago

The example in #508 shows:

apiVersion: eksctl.io/v1alpha4
kind: ClusterConfig
metadata:
  name: test-cluster-c-1
  region: eu-north-1

vpc:
  securityGroup: "sg-0f2ae54eb340e8191"
  sharedNodeSecurityGroup: "sg-02a47cee779b317a7"
  subnets:
    Private:
      eu-north-1c:
          id: "subnet-065fe8f12d0910d06"
          cidr: "10.1.128.0/19"
      eu-north-1b:
          id: "subnet-06ebc3649c1321fc5"
          cidr: "10.1.96.0/19"

iam: 
  serviceRoleARN: "arn:aws:iam::123:role/eksctl-test-cluster-a-3-cluster-ServiceRole-5YEWP7CFA24K"

nodeGroups:
  - name: ng2-private
    instanceType: m5.large
    desiredCapacity: 1
    privateNetworking: true
    securityGroups:
      withShared: true
      withLocal: false
      attachIDs: [sg-0b85ff315ea644478]
    iam:
      instanceProfileARN: "arn:aws:iam::123:instance-profile/eksctl-test-cluster-a-3-nodegroup-ng2-private-NodeInstanceProfile-Y4YKHLNINMXC"
      instanceRoleARN: "arn:aws:iam::123:role/eksctl-test-cluster-a-3-nodegroup-NodeInstanceRole-DNGMQTQHQHBJ"

After I create an eks cluster using eksctl, I can see the service role created has attached these managed policies:

And these two inline policies:

PolicyNLB has this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "elasticloadbalancing:*",
                "ec2:CreateSecurityGroup",
                "ec2:Describe*"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

while PolicyCloudWatchMetrics has this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "cloudwatch:PutMetricData"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

It has a trust policy of:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "eks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Creating a role that matches these would presumably work for the serviceRoleARN

StevenACoffman commented 5 years ago

The eksctl created nodegroup IAM role gets these managed policies:

It has three inline policies (cluster specific names truncated), depending on autoScaler and externalDNS settings:

PolicyAutoScaling:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

PolicyExternalDNSChangeSet has:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "route53:ChangeResourceRecordSets"
            ],
            "Resource": "arn:aws:route53:::hostedzone/*",
            "Effect": "Allow"
        }
    ]
}

PolicyExternalDNSHostedZones has:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "route53:ListHostedZones",
                "route53:ListResourceRecordSets"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

Presumably a role that was created in advance with these could be set for the instanceRoleARN setting.

StevenACoffman commented 5 years ago

So AFAICT, when you create a role, you get a default instance profile with the same name, so if you created a role as above, I think you can do this:

    iam:
      instanceProfileARN: "arn:aws:iam::111111:instance-profile/eks-nodes-base-role"
      instanceRoleARN: "arn:aws:iam::111111:role/eks-nodes-base-role"

I am not as acquainted with instance profiles, so if that doesn't work, I'm documenting my eksctl created instance profile while truncating the cluster specific names/ARNs, so presumably a similar, permanent, shared instance profile could be created for re-use by specifying the instanceProfileARN to match it in the config file.

$ aws iam get-instance-profile --instance-profile-name NodeInstanceProfile
{
    "InstanceProfile": {
        "Path": "/",
        "InstanceProfileName": "NodeInstanceProfile",
        "InstanceProfileId": "AIPAJYEFNR6WC4XFCA4HG",
        "Arn": "arn:aws:iam::11111:instance-profile/NodeInstanceProfile",
        "CreateDate": "2019-02-23T18:06:53Z",
        "Roles": [
            {
                "Path": "/",
                "RoleName": "NodeInstanceRole",
                "RoleId": "AROAJUB5X2XIGHLBUHRGE",
                "Arn": "arn:aws:iam::111111111:role/NodeInstanceRole",
                "CreateDate": "2019-02-23T18:06:39Z",
                "AssumeRolePolicyDocument": {
                    "Version": "2012-10-17",
                    "Statement": [
                        {
                            "Effect": "Allow",
                            "Principal": {
                                "Service": "ec2.amazonaws.com"
                            },
                            "Action": "sts:AssumeRole"
                        }
                    ]
                }
            }
        ]
    }
}
StevenACoffman commented 5 years ago

So if someone is trying to automate this in a bash script, while there are some minor additional inline policy privileges to add, this looks like it will work pretty well with slight modification

StevenACoffman commented 5 years ago

Here's what worked for me for pre-existing shared vpc and pre-existing, shared iam roles:

apiVersion: eksctl.io/v1alpha4
kind: ClusterConfig
metadata:
  name: my-test
  region: us-east-1
vpc:
  id: "vpc-11111"
  cidr: "152.28.0.0/16"
  subnets:
    private:
      us-east-1d:
          id: "subnet-1111"
          cidr: "152.28.152.0/21"
      us-east-1c:
          id: "subnet-11112"
          cidr: "152.28.144.0/21"
      us-east-1a:
          id: "subnet-11113"
          cidr: "152.28.136.0/21"
iam: 
  serviceRoleARN: "arn:aws:iam::11111:role/eks-base-service-role"
nodeGroups:
  - name: my-test-m5-private
    labels: {pool: my-test-m5-private}
    instanceType: m5.large
    desiredCapacity: 3
    minSize: 1
    maxSize: 15
    volumeSize: 50
    volumeType: gp2
    iam:
      instanceProfileARN: "arn:aws:iam::11111:instance-profile/eks-nodes-base-role"
      instanceRoleARN: "arn:aws:iam::1111:role/eks-nodes-base-role"
    privateNetworking: true
    securityGroups:
      withShared: true
      withLocal: true
      attachIDs: ['sg-11111', 'sg-11112']
    allowSSH: true
    sshPublicKeyName: 'my-instance-key'
    tags:
      'environment:basedomain': 'example.org'
errordeveloper commented 5 years ago

xref #122 #204

mumoshu commented 5 years ago

Hey!

Thanks for your efforts. This definitely helps users who wants to create cluster without huge IAM permission(that allows privilege escalation).

I'm going a bit off topic, but for simplicity, I guess we should be able to omit instanceRoleARN when instanceProfileARN is specified:

    iam:
      instanceProfileARN: "arn:aws:iam::11111:instance-profile/eks-nodes-base-role"

If it doesn't work, I'd just enhance eksctl to automatically detect the corresponding roleARN for the specified profile.

mumoshu commented 5 years ago

Also - I once considered if it would be nice to enhance eksctl to allow creating an instance profile according to the cluster config, something like:

$ eksctl utils create-instance-profile -f cluster.yaml --nodegroup my-test-m5-private
# Creates `eksctl-test-cluster-a-3-nodegroup-ng2-private-NodeInstanceProfile-Y4YKHLNINMXC` and prints the ARN of it
errordeveloper commented 5 years ago

Yusuke, I think we should consider 'eksctl create iam --cinfig-file' really, but the question is which stack it would be part of, what do you think? Maybe it should a standalone shared IAM stack that multiple clusters can import? It is kind of similar with VPC.

On Wed, 6 Mar 2019, 12:43 am KUOKA Yusuke, notifications@github.com wrote:

Also - I once considered if it would be nice to enhance eksctl to allow creating an instance profile according to the cluster config, something like:

$ eksctl utils create-instance-profile -f cluster.yaml --nodegroup my-test-m5-private

Creates eksctl-test-cluster-a-3-nodegroup-ng2-private-NodeInstanceProfile-Y4YKHLNINMXC and prints the ARN of it

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/weaveworks/eksctl/issues/573#issuecomment-469918717, or mute the thread https://github.com/notifications/unsubscribe-auth/AAPWS81occNmfWObd4auBBGxFnsCFrMQks5vTw8lgaJpZM4bOI1J .

mumoshu commented 5 years ago

Maybe it should a standalone shared IAM stack that multiple clusters can import?

A shared IAM stack per region? Sounds good!

Would it implies that we'd have a dedicated config for IAM?

# prof.yaml
apiVersion: eksctl.io/v1alpha4
kind: InstanceProfileConfig
metadata:
  name: my-test
  region: us-east-1
withAddonPolicies:
  imageBuilder: true
  autoScaler: true

Also, I was thinking if it can be a good starting point to just provide eksctl utils print-iam-policy --config-file prof.yaml? So that the user can use it to print the iam policy json that can be consumed by awscli, terraform, and cloudformation.

It is kind of similar with VPC.

Could be!

Quick thoughts - Maybe we could have a dedicated VPCConfig object that can be consumed by eksctl create vpc --config-file or eksctl utils print-vpc-stack-template --config-file. Would it share the stack with the iam related things, or not? I have no strong opinion on it yet.

kalbir commented 4 years ago

Just had a user conversation at reinvent about this 👍

pwsiegel commented 4 years ago

Our use case is that we have a pre-existing vpc, and we want developers to be able to spin up an ephemeral eks clusters inside it, but not be able to accidentally modify IAM permissions in a way that leaves us vulnerable.

I am also in this exact position, and I've been going back and forth with my AWS admin for days trying to figure out how I can have permissions to launch clusters but not hurt anything else. So this thread has been really helpful, and I offer a 👍 if there are features under consideration which would make it easier.

aclevername commented 3 years ago

Closing due to staleness. Feel free to re-open :smile: