JSON IAM Controller Policy

johnwesley commented 1 year ago

Is an existing page relevant?

What karpenter features are relevant?

Install and getting started documentation.

How should the docs be improved?

By having the karpenter-controller IAM policy, and a description of, readily available in json format for a user to review and maybe take away for a more custom install. Perhaps https://github.com/aws/karpenter/tree/main/hack/iam would be a good place?

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

ellistarn commented 1 year ago

We're wary of exposing something like this, since users will take a dependency on it and we will have an implicit contract to maintain it. We currently expose this information via the getting started guide, and are aiming to keep that as the source of truth to minimize the surface of things we need to maintain/test.

FernandoMiguel commented 1 year ago

@ellistarn i disagree. folks, like myself, community members, already try to keep up with PRs here to update https://github.com/aws-ia/terraform-aws-eks-blueprints/blob/2cd89ecc/modules/kubernetes-addons/karpenter/data.tf#L7-L25 and https://github.com/terraform-aws-modules/terraform-aws-iam/blob/c6adf42/modules/iam-role-for-service-accounts-eks/policies.tf#L508 when new policies like this appear , we have to scramble to update those.

having a source of truth in code would be helpful to refer too.

Nuru commented 1 year ago

@ellistarn I disagree, also. The Getting started guide just says to create an IRSA, it does not specify what policy to create or even give a reference to where to find the required permissions. I found this issue because the code I am using is outdated, Karpenter is getting "access denied" errors, and I'm still looking for what I need to add.

For example, the eksctl section says

eksctl create iamserviceaccount \
  --cluster "${CLUSTER_NAME}" --name karpenter --namespace karpenter \
  --role-name "${CLUSTER_NAME}-karpenter" \
  --attach-policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/KarpenterControllerPolicy-${CLUSTER_NAME}" \
  --role-only \
  --approve

but says nothing about where that KarpenterControllerPolicy-${CLUSTER_NAME}" policy came from. I need some better guidance.

ellistarn commented 1 year ago

Thanks for the feedback. This came up in working group this week too. We'll see what we can do to expose something that can be relied on.

tzneal commented 1 year ago

You can get convert the YAML to JSON:

curl https://karpenter.sh/"${KARPENTER_VERSION}"/getting-started/getting-started-with-eksctl/cloudformation.yaml | yq -o=json -

Running that on v0.1

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "Resources used by https://github.com/aws/karpenter",
  "Parameters": {
    "ClusterName": {
      "Type": "String",
      "Description": "EKS cluster name"
    }
  },
  "Resources": {
    "KarpenterNodeInstanceProfile": {
      "Type": "AWS::IAM::InstanceProfile",
      "Properties": {
        "InstanceProfileName": "KarpenterNodeInstanceProfile-${ClusterName}",
        "Path": "/",
        "Roles": [
          {
            "Ref": "KarpenterNodeRole"
          }
        ]
      }
    },
    "KarpenterNodeRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "RoleName": "KarpenterNodeRole-${ClusterName}",
        "Path": "/",
        "AssumeRolePolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Service": "ec2.${AWS::URLSuffix}"
              },
              "Action": [
                "sts:AssumeRole"
              ]
            }
          ]
        },
        "ManagedPolicyArns": [
          "arn:${AWS::Partition}:iam::aws:policy/AmazonEKS_CNI_Policy",
          "arn:${AWS::Partition}:iam::aws:policy/AmazonEKSWorkerNodePolicy",
          "arn:${AWS::Partition}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
          "arn:${AWS::Partition}:iam::aws:policy/AmazonSSMManagedInstanceCore"
        ]
      }
    },
    "KarpenterControllerPolicy": {
      "Type": "AWS::IAM::ManagedPolicy",
      "Properties": {
        "ManagedPolicyName": "KarpenterControllerPolicy-${ClusterName}",
        "PolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Resource": "*",
              "Action": [
                "ec2:CreateLaunchTemplate",
                "ec2:CreateFleet",
                "ec2:RunInstances",
                "ec2:CreateTags",
                "ec2:TerminateInstances",
                "ec2:DeleteLaunchTemplate",
                "ec2:DescribeLaunchTemplates",
                "ec2:DescribeInstances",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeImages",
                "ec2:DescribeInstanceTypes",
                "ec2:DescribeInstanceTypeOfferings",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeSpotPriceHistory",
                "ssm:GetParameter",
                "pricing:GetProducts"
              ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "iam:PassRole"
              ],
              "Resource": "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/KarpenterNodeRole-${ClusterName}"
            }
          ]
        }
      }
    }
  }
}

The JSON isn't great, but it's parsable:

curl https://karpenter.sh/"${KARPENTER_VERSION}"/getting-started/getting-started-with-eksctl/cloudformation.yaml | yq -o=json - | jq ".Resources.KarpenterControllerPolicy.Properties.PolicyDocument.Statement[].Action" | jq -s add

[
  "ec2:CreateLaunchTemplate",
  "ec2:CreateFleet",
  "ec2:RunInstances",
  "ec2:CreateTags",
  "ec2:TerminateInstances",
  "ec2:DeleteLaunchTemplate",
  "ec2:DescribeLaunchTemplates",
  "ec2:DescribeInstances",
  "ec2:DescribeSecurityGroups",
  "ec2:DescribeSubnets",
  "ec2:DescribeImages",
  "ec2:DescribeInstanceTypes",
  "ec2:DescribeInstanceTypeOfferings",
  "ec2:DescribeAvailabilityZones",
  "ec2:DescribeSpotPriceHistory",
  "ssm:GetParameter",
  "pricing:GetProducts",
  "iam:PassRole"
]

FernandoMiguel commented 1 year ago

so now all we need is a CI job that generates these JSONs and places them in git

github-actions[bot] commented 1 year ago

Labeled for closure due to inactivity in 10 days.

Nuru commented 1 year ago

@ellistarn Can you add a label or something to keep this issue from auto-closing while you work on it?

jtnz commented 1 year ago

We also disagree. We need to attach to a service account (via CDK). Here's what we currently use for cluster autoscaler.

Wouldn't the easiest solution be a AWS Managed policy?

FernandoMiguel commented 1 year ago

@Nuru it is attached to the service account

kind: ServiceAccount
apiVersion: v1
metadata:
  name: karpenter
  namespace: karpenter
  uid: 07bb3434-399c-41e0-b6cd-454f83a0a10d
  resourceVersion: "7830"
  creationTimestamp: "2022-12-08T17:27:20Z"
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::XXXXX:role/karpenter-irsa
secrets:
- name: karpenter-token-bgvgm
automountServiceAccountToken: true

stevehipwell commented 1 year ago

As I don't use eksctl or CloudFormation the current getting started docs aren't a lot of use to me, I distinctly remember that the docs used to be more comprehensive than this?

I would expect the primary documentation to use the basic tooling with extended docs for additional tools (Terraform would likely have a greater audience than eksctl and definitely greater than CloudFormation); so the aws CLI and kubectl should be the only prerequisites. IMHO as eksctl used for IRSA removes control for the ServiceAccount from the Helm chart this should be called out as an anti-pattern used for convenience.

The hard requirements for consuming component like this would be as follows.

CRDs in repo to be referenced by release tag (this is good)
IAM policy as JSON in repo to be referenced by release tag (I can't see this)
Minimum required steps (some of the steps are not actually required if provisioners are configured)

chrisnegus commented 10 months ago

We added a new page that details the cloudformation.yaml used in the Karpenter Getting Started guide. It will be published with v0.32, but is currently available in preview here: Cloudformation. This should help if you want to use tools other than CloudFormation to set up Karpenter controller permissions. I'm going to close this issue, but feel free to reopen it if there is more that you would like to see in this area.

Nuru commented 10 months ago

@chrisnegus Thank you for the documentation. The explanations of why Karpenter needs what permissions, and commentary on what the permissions actually grant, are very helpful.

Nevertheless, I'd ask you to re-open this issue because we are still missing what was originally requested: the required IAM Polic(y/ies) in JSON format. All the official AWS documentation provides policies in JSON format, so tools have a well-documented way of creating policies from JSON. People not already using Cloudformation do not want to add Cloudformation to their tool chain in order to create a policy.

Currently I am piecing together the policy to use in Terraform from the source code maintained by terraform-aws-modules, but I do not like relying on a third party to provide security policies like this. Plus that code has variables interpolated that I do not have set, so I cannot automatically update my code based on updates to that code.

For comparison, consider the AWS Load Balancer Controller project, which provides links to downloadable JSON policies in their documentation. While I would prefer that the policy be available via the Helm chart, this is at least something I can easily automate without needing to use a new tool just for that.

njtran commented 10 months ago

Hey @Nuru, looks like the initial block for the controller policy is in CFN, but each of the required policies are in JSON. Are you able to conjoin these together to form your JSON policy? The policies are kept separate so that users are aware of what each policy is needed for. Is this sufficient?

Nuru commented 10 months ago

Hey @Nuru, looks like the initial block for the controller policy is in CFN, but each of the required policies are in JSON. Are you able to conjoin these together to form your JSON policy? The policies are kept separate so that users are aware of what each policy is needed for. Is this sufficient?

No, for a few reasons. First off, the policies still have CFN parameters in them like ${AWS::Region}, so I cannot use these directly. Second, I cannot easily download them.

The goal is to be able to automatically update the policy when upgrading to a later version of Karpenter that needs some policy change. In general, for practically every other component I am familiar with, non-breaking upgrades can be handled completely automatically by using tools like Renovate in conjunction with other tools like Helm and Terraform. Even with this preview documentation, future upgrades of Karpetner are going to require manual intervention to read the documentation, parse the policy changes, and manually apply the changes to whatever tool configuration someone is using.

It is great that the policies are so well documented and explained. In this regard, Karpenter is ahead of most other projects. But as a separate matter, long-term maintenance of installations is just much harder if the policies are not available in formats suitable for automation to apply and update.

jonathan-innis commented 8 months ago

I aded some commentary in #5309 but relaying what I mentioned over there in this issue

There's a trade-off here for us of having to maintain and keep two different types of documents consistent here. Are you able to convert from the current Getting Started CloudFormation guide to the JSON document that you need? There was a similar discussion here on this topic https://github.com/aws/karpenter-provider-aws/issues/2649, so if this issue boils down to needing a JSON representation of the IAM policy, I'd point you to the other issue to add additional input.

We're definitely not strictly opposed to it. I'd like us to support better ways to automate this for users but the problem right now is that we don't have automation that allows us to maintain both the JSON policy and the CloudFormation documents that we need. If we had that in place and had a single source-of-truth for maintaining the JSON document, then I'd be all on-board.

We'd happily accept a contribution that supported some way to achieve this; most of the maintainers don't currently have the bandwidth to support this right now.

ksquaredkey commented 8 months ago

Continuing a conversation started in #5407 and #5309,

If anything, the CloudFormation file should REFERENCE that file to get the canonical policy value, keeping the single source of truth

Agree, as mentioned in https://github.com/aws/karpenter-provider-aws/issues/2649, we'd definitely accept a change to our docs that pointed us in that direction, so long as we can make sure to have a single place where the policy is listed 👍🏼

I'm not a CloudFormation user (see earlier posts) but from a quick read of the docs it looks like the only thing we can do to get the CloudFormation to reference a file is the Fn::Transform using an AWS::Include and that the AWS::Include "location:" syntax only supports an S3 URI (it explicitly says GitHub repo is not supported - boo!!).

Does the project have an AWS S3 bucket associated with it that we could post the latest canonical policy file in to make it the single source of truth for the JSON policy? Or a way to get CloudFormation to include from a regular URL? Of the various automations - Terraform, CDK, etc - it looks like CloudFormation is the most restrictive in its include syntax to reference the IAM policy. It seems a strange choice for the example deployment.

felipewnp commented 3 weeks ago

Plus one on this.

It's really hard to understand why there still aren't pure JSON policies. Since 2022.

It's far easier for someone well-versed in Cloud Formation to automate the translation from JSON to CFN than someone who never saw CFN to translate that to JSON.

Trying to automate the extraction of the JSON part of the CFN is also way more complicated than inserting the same JSON into it.

If available hands is the problem and the below policy is (somewhat) correct, I can send a PR to send this file somewhere to address this...

## VARIABLES:
AWS_REGION=
CLUSTER_NAME=
AWS_PARTITION=
AWS_ACCOUNT_ID=
KARPENTER_NODE_ROLE_ARN=
KARPENTER_INTERRUPTION_QUEUE_ARN=

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowScopedEC2InstanceAccessActions",
      "Effect": "Allow",
      "Resource": [
        "arn:${AWS_PARTITION}:ec2:${AWS_REGION}::image/*",
        "arn:${AWS_PARTITION}:ec2:${AWS_REGION}::snapshot/*",
        "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:security-group/*",
        "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:subnet/*"
      ],
      "Action": [
        "ec2:RunInstances",
        "ec2:CreateFleet"
      ]
    },
    {
      "Sid": "AllowScopedEC2LaunchTemplateAccessActions",
      "Effect": "Allow",
      "Resource": "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:launch-template/*",
      "Action": [
        "ec2:RunInstances",
        "ec2:CreateFleet"
      ],
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned"
        },
        "StringLike": {
          "aws:ResourceTag/karpenter.sh/nodepool": "*"
        }
      }
    },
    {
      "Sid": "AllowScopedEC2InstanceActionsWithTags",
      "Effect": "Allow",
      "Resource": [
        "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:fleet/*",
        "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:instance/*",
        "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:volume/*",
        "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:network-interface/*",
        "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:launch-template/*",
        "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:spot-instances-request/*"
      ],
      "Action": [
        "ec2:RunInstances",
        "ec2:CreateFleet",
        "ec2:CreateLaunchTemplate"
      ],
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned",
          "aws:RequestTag/eks:eks-cluster-name": "${CLUSTER_NAME}"
        },
        "StringLike": {
          "aws:RequestTag/karpenter.sh/nodepool": "*"
        }
      }
    },
    {
      "Sid": "AllowScopedResourceCreationTagging",
      "Effect": "Allow",
      "Resource": [
        "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:fleet/*",
        "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:instance/*",
        "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:volume/*",
        "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:network-interface/*",
        "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:launch-template/*",
        "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:spot-instances-request/*"
      ],
      "Action": "ec2:CreateTags",
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned",
          "aws:RequestTag/eks:eks-cluster-name": "${CLUSTER_NAME}",
          "ec2:CreateAction": [
            "RunInstances",
            "CreateFleet",
            "CreateLaunchTemplate"
          ]
        },
        "StringLike": {
          "aws:RequestTag/karpenter.sh/nodepool": "*"
        }
      }
    },
    {
      "Sid": "AllowScopedResourceTagging",
      "Effect": "Allow",
      "Resource": "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:instance/*",
      "Action": "ec2:CreateTags",
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned"
        },
        "StringLike": {
          "aws:ResourceTag/karpenter.sh/nodepool": "*"
        },
        "StringEqualsIfExists": {
          "aws:RequestTag/eks:eks-cluster-name": "${CLUSTER_NAME}"
        },
        "ForAllValues:StringEquals": {
          "aws:TagKeys": [
            "eks:eks-cluster-name",
            "karpenter.sh/nodeclaim",
            "Name"
          ]
        }
      }
    },
    {
      "Sid": "AllowScopedDeletion",
      "Effect": "Allow",
      "Resource": [
        "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:instance/*",
        "arn:${AWS_PARTITION}:ec2:${AWS_REGION}:*:launch-template/*"
      ],
      "Action": [
        "ec2:TerminateInstances",
        "ec2:DeleteLaunchTemplate"
      ],
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned"
        },
        "StringLike": {
          "aws:ResourceTag/karpenter.sh/nodepool": "*"
        }
      }
    },
    {
      "Sid": "AllowRegionalReadActions",
      "Effect": "Allow",
      "Resource": "*",
      "Action": [
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeImages",
        "ec2:DescribeInstances",
        "ec2:DescribeInstanceTypeOfferings",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeLaunchTemplates",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSpotPriceHistory",
        "ec2:DescribeSubnets"
      ],
      "Condition": {
        "StringEquals": {
          "aws:RequestedRegion": "${AWS_REGION}"
        }
      }
    },
    {
      "Sid": "AllowSSMReadActions",
      "Effect": "Allow",
      "Resource": "arn:${AWS_PARTITION}:ssm:${AWS_REGION}::parameter/aws/service/*",
      "Action": "ssm:GetParameter"
    },
    {
      "Sid": "AllowPricingReadActions",
      "Effect": "Allow",
      "Resource": "*",
      "Action": "pricing:GetProducts"
    },
    {
      "Sid": "AllowInterruptionQueueActions",
      "Effect": "Allow",
      "Resource": "${KARPENTER_INTERRUPTION_QUEUE_ARN}",
      "Action": [
        "sqs:DeleteMessage",
        "sqs:GetQueueUrl",
        "sqs:ReceiveMessage"
      ]
    },
    {
      "Sid": "AllowPassingInstanceRole",
      "Effect": "Allow",
      "Resource": "${KARPENTER_NODE_ROLE_ARN}",
      "Action": "iam:PassRole",
      "Condition": {
        "StringEquals": {
          "iam:PassedToService": "ec2.amazonaws.com"
        }
      }
    },
    {
      "Sid": "AllowScopedInstanceProfileCreationActions",
      "Effect": "Allow",
      "Resource": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:instance-profile/*",
      "Action": [
        "iam:CreateInstanceProfile"
      ],
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned",
          "aws:RequestTag/eks:eks-cluster-name": "${CLUSTER_NAME}",
          "aws:RequestTag/topology.kubernetes.io/region": "${AWS_REGION}"
        },
        "StringLike": {
          "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*"
        }
      }
    },
    {
      "Sid": "AllowScopedInstanceProfileTagActions",
      "Effect": "Allow",
      "Resource": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:instance-profile/*",
      "Action": [
        "iam:TagInstanceProfile"
      ],
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned",
          "aws:ResourceTag/topology.kubernetes.io/region": "${AWS_REGION}",
          "aws:RequestTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned",
          "aws:RequestTag/eks:eks-cluster-name": "${CLUSTER_NAME}",
          "aws:RequestTag/topology.kubernetes.io/region": "${AWS_REGION}"
        },
        "StringLike": {
          "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*",
          "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*"
        }
      }
    },
    {
      "Sid": "AllowScopedInstanceProfileActions",
      "Effect": "Allow",
      "Resource": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:instance-profile/*",
      "Action": [
        "iam:AddRoleToInstanceProfile",
        "iam:RemoveRoleFromInstanceProfile",
        "iam:DeleteInstanceProfile"
      ],
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned",
          "aws:ResourceTag/topology.kubernetes.io/region": "${AWS_REGION}"
        },
        "StringLike": {
          "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*"
        }
      }
    },
    {
      "Sid": "AllowInstanceProfileReadActions",
      "Effect": "Allow",
      "Resource": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:instance-profile/*",
      "Action": "iam:GetInstanceProfile"
    },
    {
      "Sid": "AllowAPIServerEndpointDiscovery",
      "Effect": "Allow",
      "Resource": "arn:${AWS_PARTITION}:eks:${AWS_REGION}:${AWS_ACCOUNT_ID}:cluster/${CLUSTER_NAME}",
      "Action": "eks:DescribeCluster"
    }
  ]
}

bryantbiggs commented 3 weeks ago

IAM policies should be treated as a versioned artifact since the permissions are tied directly to the underlying implementation. I also want to clarify that this is not just an issue specific Karpenter, but for all projects (controllers) that interact with AWS APIs.

If I hypothetically (crudely) re-create something like Karpenter - as the project grows and evolves, adding new functionality, the associated IAM permissions evolve along with the underlying implementation. As Karpenter has evolved, from adding spot interruption native support to creating IAM instance profiles on behalf of users - these are implementation details that are tied directly to a set of required IAM permissions. However, these details are not readily known through the common version update mechanisms - users are required to navigate back to the core project and specifically the Getting Started.. section (provided you know to look there) to find an example IAM policy. I believe the ask in this issue is to elevate this getting started details to a first class versioned artifact that is directly consumable via an API. The simplest and easiest way to satisfy this would be to simply create a JSON file of the policy in the *AWS provider repo that can be consumed via a standard HTTP request. Doing this will now mean that as new git tags are created on the AWS provider repository, users have a the normal means to reference the artifact within the repository using either the git tag or git sha. Now when users upgrade from something like v0.1.0 to v0.2.0 - the IAM permissions used by Karpenter controller v0.2.0 match what has been tested and validated by the project itself.

There are caveats and footguns with this approach - namely the use of variables to tailor the policy to a users account, region, cluster, etc. I don't have a good solution for this - its possible to do templating but this could face challenges if the variables used in the upstream policy are changed, as well as how to discover or learn about new variables that have been added. I started exploring this concept with the PoC projects below but abandoned due to this brittleness around finding a way to create a variable "contract" of sorts.

https://github.com/clowdhaus/example-external-policies-consumer - consumes the externally provided policy https://github.com/clowdhaus/example-external-policies - example of a "project" that provides its respective policy that is versioned alongside the software that requires the permissions

I am providing this information to highlight the fact that the request while valid, there aren't any known viable solutions at this time. However, perhaps the capacity of the community at large might come up with alternatives that are viable that could be considered!

[Edit 1] CloudFormation is not a suitable medium because that is specific to one tool/framework. The policy should be tooling/framework agnostic so that it can be consumed by all regardless of what they use

stevehipwell commented 3 weeks ago

@bryantbiggs I think a reference JSON policy should be provided even if there is no standard way to template as the diff between versions, as it would still at least significantly reduce the cognitive load to process changes between versions. This could easily be augmented to support templating by representing the template variables (e.g. partition, account, region, cluster name, etc) by know values in a similar way to how date formats in Go work.

[Edit 1] CloudFormation is not a suitable medium because that is specific to one tool/framework. The policy should be tooling/framework agnostic so that it can be consumed by all regardless of what they use

Having a single CloudFormation file, as is used by Karpenter, makes it harder to detect IAM changes due to the additional noise from unrelated changes.

aws / karpenter-provider-aws