aws.tags on karpenter controller/helm chart values

fonteslucas commented 7 months ago

Description

What problem are you trying to solve?

Tag EC2 Nodes deployed by karpenter without specify in NodeClasses

How important is this feature to you? We use a strategy where deploy and install EKS and a bunch of addons using terraform and ci/cd pipelines. With previous versions of Karpenter we used to input in settings.aws.tags. This save us alot of time and ensure that EC2 Node will be created with required tags defined in or organizations.

This was great, because we use ArgoCD to deploy not only applications, but manage Karpenter resources to.

Without tags configuration in helm values, we need to trust our devs to input this tags, that today is enforced automatically when configuring the Karpenter helm values

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

jonathan-innis commented 7 months ago

Tag EC2 Nodes deployed by karpenter without specify in NodeClasses

We used to have this feature prior to v0.32.0 and then dropped it because we assumed that all users could achieve the same thing with templating, rather than having to create global resources for Karpenter. Is there anything in your environment that makes templating those "default" tags hard?

fonteslucas commented 7 months ago

Is there anything in your environment that makes templating those "default" tags hard?

I'll try to explain the process and context of deploy, because I think that will clarify

We have one team that manage and maintiain a bunch of terraform modules, and one of the most importants is the EKS Terraform module. It deploy not only the infrasctructure of EKS, but also a bunch of addons.

All the resources are deployed using CI/CD strategy with AzureDevOps, and each pipeline, inside an AzureDevOps Project, has a library of the tags that is inputed in the resources during the creation.

With this strategy, we ensure that resources created in the AWS Accounts recieve tags, without need to use SCP + Tag Policies strategy to block resource creation without required tags.

The "problem" is the projects are managed by other teams and areas, that not ours . And check that teams will templating those default tags in NodeClasses is a bit of hard-working for us. The templates are deployed using ArgoCD. ArgoCD look to a repository inside the AzureDevOps Project that hold the applications.

Today, the areas only need to inform like, "which ec2 instance type I need" in the templates, and all the tags are inputed automatically, because the the helm_config of karpenter addon is pre configured.

The part of terraform module that "recieve" the tags.

helm_config = merge(
    {
      name       = local.name
      chart      = local.name
      repository = "oci://public.ecr.aws/karpenter"
      version    = "v0.30.0"
      namespace  = local.name
      values = [
        <<-EOT
          settings:
            aws:
              clusterName: ${var.cluster_name}
              clusterEndpoint: ${var.cluster_endpoint}
              defaultInstanceProfile: ${var.node_iam_instance_profile}
              interruptionQueueName: ${try(aws_sqs_queue.this[0].name, "")}
              tags: ${local.tags} ######<- here is where we configure the tags "globally" on the controller and inputed by the pipeline
        EOT
      ]
      description = "karpenter Helm Chart for Node Autoscaling"
    },
    var.helm_config
  )

I mean, I don't know how hard it is, or if it's possible to bring back the possibility to inform global tags with optional configuration.

This "problem" maybe can occurs in the defaultInstanceProfile global settings, because removing from there, we need to delegate to teams/area to inform what role they want to use, and, somehow it will be not mapped in the aws-auth. I need to test.

    {
      groups   = ["system:bootstrappers", "system:nodes"]
      rolearn  = try("${aws_iam_role.this[0].arn}", "")
      username = "system:node:{{EC2PrivateDNSName}}"
    }

Thanks in advance :)

jonathan-innis commented 6 months ago

If you need to ensure that the users for the clusters that you are managing are setting up their EC2NodeClasses in the correct way, have you looked at using some kind of policy agent to manage this enforcement? One of the primary reasons that we dropped this was that we thought that the same mechanisms that were used at the process-level here could also be enforced by other mechanisms in Kubernetes, like policy agents and validating webhooks/policies.

See: https://kyverno.io/ and https://www.openpolicyagent.org/

fonteslucas commented 6 months ago

Well, I'm testing kyverno addon and Cluster Policy to input this informations (tags and instanceProfile) in the resources without need to "delegate" to the area , but now I'm facing the same issue spcified here:

https://github.com/aws/karpenter-provider-aws/issues/5796

sidewinder12s commented 2 weeks ago

I think the flip from just let me handle this for my users to I now need to figure out more policy enforcement to yell about yet another setting someone needs to manage is what makes this change painful.

At a minimum, it is extremely common to apply global tags to resources like Cluster or Contact that I want applied to everything karpenter is generating from a cost and governance perspective.

Now I need to go figure out how to plumb that value into multiple CRDs in every invocation of that CRD which are likely to be in multiple different deployment pipelines for something like Karpenter NodePool configurations

aws / karpenter-provider-aws

aws.tags on karpenter controller/helm chart values #5860

Description