Feature Request: equivalent of `kubectl patch`

[zapman449]

Terraform Version

Terraform v0.12.18

Affected Resource(s)

n/a (request for new resource)

In AWS EKS, clusters come "pre-configured" with several things running in the kube-system namespace. We need to patch those pre-configured things, while retaining any "upstream" changes which happen to be made. (for example: set HTTP_PROXY variables)

kubectl provides the patch keyword to handle this use-case.

The kubernetes provider for terraform should do the same.

Proposed example (this would add the proxy-environment-variables ConfigMap to the existing envFrom list which already contains aws-node-environment-variable-additions for the container named aws-node):

resource "kubernetes_patch" "aws-node" {
  kind = daemonset
  metadata {
    name      = "aws-node"
    namespace = "kube-system"
  }
  spec {
    template {
      spec {
        container {
          name = "aws-node"
          envFrom {
            [
              configMapRef {
                name: proxy-environment-variables
              }
              configMapRef {
                name: aws-node-environment-variable-additions
              }
            ]
          }
        }
      }
    }
  }
}

[antonosmond]

I have 2 additional use cases for the same feature, both on EKS.

1. If you want to utilise node taints & tolerations for all your nodes, any EKS managed k8s resources e.g. coredns must be patched to tolerate the taints.

2. Fargate on EKS. If you want to run a nodeless cluster and use Fargate to run everything, some EKS managed resources e.g. coredns prevent this via an annotation e.g.

   annotations:
   eks.amazonaws.com/compute-type: ec2

   The annotation must be removed. The ability to patch resources would solve both these use cases and many others.

[oleksandrsemak]

I have 2 additional use cases for the same feature, both on EKS.

1. If you want to utilise node taints & tolerations for all your nodes, any EKS managed k8s resources e.g. coredns must be patched to tolerate the taints.
2. Fargate on EKS. If you want to run a nodeless cluster and use Fargate to run everything, some EKS managed resources e.g. coredns prevent this via an annotation e.g.

  annotations:
    eks.amazonaws.com/compute-type: ec2

The annotation must be removed. The ability to patch resources would solve both these use cases and many others.

Yeah also it would be nice to have equivalent of kubectl taint node as it will not work w/o taint node before

[stoimendhristov]

+1

[stoimendhristov]

I am currently trying to update an existing ConfigMap and simply add more rules to it but once the CM created it seems that it cannot be referred to in order to be updated.

Any thoughts?

Thanks

[adilc]

+1

[Sharathmk99]

+1

[eugene-burachevskiy]

When we setup EKS cluster with terraform and are using tainted on-demand nodes for all system services, we have to patch CoreDNS first to make all further installed apps working. For now we can't patch existing EKS CoreDNS with terraform so we have to install 3rd party CoreDNS helm chart at the beginning.

Ability to patch existing deployments would be really great.

[hijakebye]

+1 Would love this for some of our enterprise EKS Fargate Deployments

[ivan-sukhomlyn]

It would nice to have such a Terraform resource to patch EKS aws-node DaemonSet with a custom ServiceAccount. For example, in the case of the IRSA approach usage for Pods authorization.

[vide]

This is also needed to patch EKS clusters hit by https://github.com/kubernetes/kubernetes/issues/61486

[blawlor]

This feature would feed very well into things like custom CNI on EKS

[memory]

This would also help for management of service meshes such as linkerd or istio, where one might want to add annotations to control mesh proxy injection into the kube-system or default namespace.

This request is actually being made in different forms in several issues now, see also:

https://github.com/hashicorp/terraform-provider-kubernetes/issues/238 https://github.com/hashicorp/terraform/issues/22754

[memory]

For anyone else who's running into this, we've for the moment worked around it with a truly awful abuse of the null resource and local provisioner:

resource "null_resource" "k8s_patcher" {
  triggers = {
    // fire any time the cluster is update in a way that changes its endpoint or auth
    endpoint = google_container_cluster.default.endpoint
    ca_crt   = google_container_cluster.default.master_auth[0].cluster_ca_certificate
    token    = data.google_client_config.provider.access_token
  }

  # download kubectl and patch the default namespace
  provisioner "local-exec" {
    command = <<EOH
cat >/tmp/ca.crt <<EOF
${base64decode(google_container_cluster.default.master_auth[0].cluster_ca_certificate)}
EOF
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && chmod +x ./kubectl
./kubectl \
  --server="https://${google_container_cluster.default.endpoint}" \
  --token="${data.google_client_config.provider.access_token}" \
  --certificate_authority=/tmp/ca.crt \
  patch namespace default \
  -p '{"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{"linkerd.io/inject":"enabled"},"name":"default"}}'
EOH
  }
}

[Haptr3c]

I tweaked @memory's null_resource workaround to work with the aws provider. This should save anyone looking to run fargate-only EKS a bit of time.

resource "aws_eks_fargate_profile" "coredns" {
  cluster_name           = aws_eks_cluster.main.name
  fargate_profile_name   = "coredns"
  pod_execution_role_arn = aws_iam_role.fargate_pod_execution_role.arn
  subnet_ids             = var.private_subnets.*.id
  selector {
    namespace = "kube-system"
    labels = {
      k8s-app = "kube-dns"
    }
  }
}

resource "null_resource" "k8s_patcher" {
  depends_on = [ aws_eks_fargate_profile.coredns ]
  triggers = {
    // fire any time the cluster is update in a way that changes its endpoint or auth
    endpoint = aws_eks_cluster.main.endpoint
    ca_crt   = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
    token    = data.aws_eks_cluster_auth.cluster.token
  }
  provisioner "local-exec" {
    command = <<EOH
cat >/tmp/ca.crt <<EOF
${base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)}
EOF
apk --no-cache add curl && \
curl -o aws-iam-authenticator https://amazon-eks.s3.us-west-2.amazonaws.com/1.17.9/2020-08-04/bin/linux/amd64/aws-iam-authenticator && chmod +x ./aws-iam-authenticator && \
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && chmod +x ./kubectl && \
mkdir -p $HOME/bin && mv ./aws-iam-authenticator $HOME/bin/ && export PATH=$PATH:$HOME/bin && \
./kubectl \
  --server="${aws_eks_cluster.main.endpoint}" \
  --certificate_authority=/tmp/ca.crt \
  --token="${data.aws_eks_cluster_auth.cluster.token}" \
  patch deployment coredns \
  -n kube-system --type json \
  -p='[{"op": "remove", "path": "/spec/template/metadata/annotations/eks.amazonaws.com~1compute-type"}]'
EOH
  }
}

[Z3R6]

Any updates?

[kvaidas]

The following might also be a viable workaround:

resource "local_file" "kubeconfig" {
  filename = pathexpand("~/.kube/config")
  content = <<-CONFIG
    apiVersion: v1
    kind: Config
    clusters:
    - name: clustername
      cluster:
        server: ${aws_eks_cluster.this.endpoint}
        certificate-authority-data: ${aws_eks_cluster.this.certificate_authority.0.data}
    contexts:
    - name: contextname
      context:
        cluster: clustername
        user: username
    current-context: contextname
    users:
    - name: username
      user:
        token: ${data.aws_eks_cluster_auth.this-auth.token}
  CONFIG
}

Might work quicker since the token should only be requested once and then reused for any kubectl commands.

Also doesn't depend on having aws-cli installed.

[vdahmane]

Any update?

[johanferguth]

Same question for me :) : If I want to allow other users than me to manage an Aws Eks cluster, I have to edit the configmap aws-auth. It could be very useful to patch this configmap after a deployment rather than replace it totally.

[ams0]

Adding to the list, patching argocd-cm ConfigMap to add a private repository. I bootstrap AKS+ArgoCD and I'd like to use a private repos for the apps.

[GolubevV]

Same issue here, with the need to patch the coredns for taints-tollerations setup and aws-node daemonset for some parameters tweaking (like IP warm target and external SNAT enable).

Will be really nice to be able to get all resources provisioned in one shot by terraform without workarrounds like local-exec provisioner which does not work on TFE out of the box due to missing kubectl.

[holgerson97]

This is also relevant when you want to deploy an EKS cluster only running Fargate. you need to patch the existing CoreDNS deployment in order to deploy it as Fargate.

[shanoor]

Also needed to simply edit the coredns-custom configmap that is created by default in AKS.

[adamrushuk]

Adding to the list, patching argocd-cm ConfigMap to add a private repository. I bootstrap AKS+ArgoCD and I'd like to use a private repos for the apps.

I've got a similar requirement, so until there is a better method, I'm using a template and null resource:

# argocd-cm patch
# https://registry.terraform.io/providers/hashicorp/template/latest/docs/data-sources/file
data "template_file" "argocd_cm" {
  template = file(var.argocd_cm_yaml_path)
  vars = {
    tenantId    = data.azurerm_client_config.current.tenant_id
    appClientId = azuread_service_principal.argocd.application_id
  }
}

# https://www.terraform.io/docs/provisioners/local-exec.html
resource "null_resource" "argocd_cm" {
  triggers = {
    yaml_contents = filemd5(var.argocd_cm_yaml_path)
    sp_app_id     = azuread_service_principal.argocd.application_id
  }

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]
    environment = {
      KUBECONFIG = var.aks_config_path
    }
    command = <<EOT
      kubectl patch configmap/argocd-cm --namespace argocd --type merge --patch "${data.template_file.argocd_cm.rendered}"
    EOT
  }

  depends_on = [
    local_file.kubeconfig,
    null_resource.argocd_configure
  ]
}

[eahrend]

Adding a use case to this list as well, I'm on GKE and I'm using the terraform code below to create a namespace/secrets to help bootstrap a cluster with service account keys required by the applications. If the secret data changes, I'd like to overwrite the secret, but this fails due to the namespace existing already.

resource "kubernetes_namespace" "default" {
  metadata {
    name = var.namespace
  }
}

resource "kubernetes_secret" "default" {
  depends_on = [kubernetes_namespace.default]
  metadata {
    name = var.kube_secret_name
    namespace = var.namespace
  }
  data = var.secret_data
  type = var.secret_type
}

[jrhouston]

Thanks everyone for your patience on this issue.

We're looking at implementing this feature and are discussing where it should live and what its implementation is going to look like. If you want to add your 2¢ please feel free to contribute your thoughts to this proposal PR: https://github.com/hashicorp/terraform-provider-kubernetes/pull/1257

[bfelaco]

+1 on this feature request for a generic way to apply patches to existing resources. It would also be helpful to have a prescriptive way to handle the common requirement to create/update specific key/values within an existing ConfigMap or Secret.

[alexmnyc]

Please note ability to add annotation to existing resources like service accounts installed by managed services like EKS, there should be

resource "kubernetes_service_account" {
    allow_import_if_exists = true
    metadata {
       name = "aws-node"
       namespace = "kube-system"
       # https://docs.aws.amazon.com/eks/latest/userguide/cni-iam-role.html
       eks.amazonaws.com/role-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:role/<AmazonEKSCNIRole>
   }
}

https://github.com/hashicorp/terraform-provider-kubernetes/issues/692

[jrhouston]

Thanks for adding your thoughts @bfelaco @alexmnyc.

I'm collecting acceptance test criteria for this feature. If you have a specific use case you need this for please share with as much detail as possible and add here to the proposal PR linked above. If you are already solving it outside or Terraform or with a null_resource I'd love to hear about that too.

[herrLierb]

deamon-set-patch.zip

I can share a sample using null-resources to enable IRSA for the node daemonset as described here: https://aws.github.io/aws-eks-best-practices/security/docs/iam/#update-the-aws-node-daemonset-to-use-irsa

many thanks to @jensbac ! :)

[mbevc1]

Adding myself to notification :+1: this one

[RichiCoder1]

Adding myself to notification 👍 this one

For future reference, you can just "Subscribe" to the issue.

[Sebor]

One more use case of AWS EKS: there is a default storage class "gp2" when I create cluster. But during cluster provisioning I create another storage class ("gp3", for example) and I want to set it as default. To do that firstly I have to set annotation "storageclass.kubernetes.io/is-default-class: false" for "gp2" and only after that create new storage class with annotation "storageclass.kubernetes.io/is-default-class: true".

Currently I do that via local-provisioner:

  provisioner "local-exec" {
    command = "kubectl patch storageclass gp2 -p '{\"metadata\": {\"annotations\":{\"storageclass.kubernetes.io/is-default-class\":\"false\"}}}'"

    environment = {
      KUBECONFIG = module.eks_cluster.kubeconfig_filename
    }
  }

It's ugly and I must have local kubeconfig file

[scalen]

Flagging the details of another usecase in AWS EKS: enabling CNI support for Pod-level Security Groups/secure networking. This requires setting an environment variable in the template of the aws-node DaemonSet, in the kube-system namespace.

Ideally, this would remain managed EKS in general, and just have that environment variable changed by Terraform.

[aleixrm-scopely]

Another use case, similar to @scalen 's one, is patching the aws-cni-plugin add-on for changing the WARM_ENI_TARGET parameter setting an environment variable (https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/eni-and-ip-target.md)

[aavileli]

This is such an important feature. It will compliment kubernetes_manifest resource.

[artificial-aidan]

Added a bit of configurability to my patch resource

data "aws_eks_cluster" "cluster" {
  name = var.cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = var.cluster_name
}

resource "local_file" "config" {
  content  = ""
  filename = "${path.module}/.terraform/config.yaml"
}

resource "local_file" "kube_ca" {
  content  = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
  filename = "${path.module}/.terraform/ca.crt"
}

resource "null_resource" "patch_cni" {
  triggers = {
    always_run = "${timestamp()}"
  }

  provisioner "local-exec" {
    command = "kubectl set env ds aws-node --server=$KUBESERVER --token=$KUBETOKEN --certificate-authority=$KUBECA -n kube-system ENABLE_PREFIX_DELEGATION=true WARM_PREFIX_TARGET=1"

    environment = {
      KUBECONFIG = local_file.config.filename
      KUBESERVER = data.aws_eks_cluster.cluster.endpoint
      KUBETOKEN  = data.aws_eks_cluster_auth.cluster.token
      KUBECA     = local_file.kube_ca.filename
    }
  }
}

[jkroepke]

Maybe I could understand the server-side apply feature wrong.

A equivalent of kubectl patch could be a partial apply with the kubernetes_manifest resource that only updates the fields defined inside the resource. In combination with the kubernetes service-side apply feature, only the fields should be update managed by the field-manager terraform.

[scalen]

A equivalent of kubectl patch could be a partial apply with the kubernetes_manifest resource that only updates the fields defined inside the resource. In combination with the kubernetes service-side apply feature, only the fields should be update managed by the field-manager terraform.

@jkroepke This is definitely true, in some contexts, but it seems to depend on how the Server-Side Apply features are invoked. For example, it isn't clear (to me) from the documentation if incomplete resources are supported via all channels. (Rhetorical questions) Does the current behaviour of kubernetes_manifest (and/or the other kubernetes_* resources) support such incomplete manifests? Or is there an argument available to set to support partial applies?

Either way, clarification of these in the documentation would be very useful!

[jkroepke]

A second approach would be a new resource like kubernetes_manifest_field that manages only one field of an existing resource. That could be the more terraform way.

[Maxwell2022]

With the null_resource work around, I found that using kubeconfig template looks a bit more clean (source)

data "template_file" "kubeconfig" {
  template = <<-EOF
    apiVersion: v1
    kind: Config
    current-context: terraform
    clusters:
    - name: ${aws_eks_cluster.cluster.name}
      cluster:
        certificate-authority-data: ${aws_eks_cluster.cluster.certificate_authority.0.data}
        server: ${aws_eks_cluster.cluster.endpoint}
    contexts:
    - name: terraform
      context:
        cluster: ${aws_eks_cluster.cluster.name}
        user: terraform
    users:
    - name: terraform
      user:
        token: ${data.aws_eks_cluster_auth.cluster.token}
  EOF
}

resource "null_resource" "update_coredns_annotations" {
  triggers = {
    kubeconfig = base64encode(data.template_file.kubeconfig.rendered)
    cmd_patch = <<-EOF
      cat <<YAML | kubectl \
        -n kube-system \
        --kubeconfig <(echo $KUBECONFIG | base64 --decode) \
        patch deployment coredns \
        --type json \
        -p='[{"op": "remove", "path": "/spec/template/metadata/annotations/eks.amazonaws.com~1compute-type"}]'
    EOF
  }

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]
    environment = {
      KUBECONFIG = self.triggers.kubeconfig
    }
    command = self.triggers.cmd_patch
  }

  depends_on = [aws_eks_fargate_profile.coredns, aws_eks_cluster.cluster]
}

[jkroepke]

interesting, but not cross os compatible.

[robertb724]

Here is the workaround we came up with, this will run as soon as the cluster is created in our case:

resource "kubernetes_job" "kube_system_labeler" {
  metadata {
    name      = "kube-system-labeler"
    namespace = utility
  }
  spec {
    template {
      metadata {}
      spec {
        service_account_name = utility
        container {
          name    = "ns-labeler"
          image   = "bitnami/kubectl:latest"
          command = ["/bin/sh", "-c", "kubectl label ns kube-system --overwrite config.linkerd.io/admission-webhooks=disabled admission.gatekeeper.sh/ignore=no-self-managing"]
        }
        restart_policy = "Never"
      }
    }
  }
  wait_for_completion = true
  timeouts {
    create = "5m"
  }
}

[watsonjm]

This is an issue for us as well since we frequently do work in AWS EKS where other users need to be added to aws-auth configmap, but this is not currently possible without external dependencies (kubectl).

[shinebayar-g]

@watsonjm fyi you can manually add aws-auth configmap before it's created automatically for you. You have to create this configmap in terraform before you create your first nodepool. https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html

[marcuz]

This is an issue for us as well since we frequently do work in AWS EKS where other users need to be added to aws-auth configmap, but this is not currently possible without external dependencies (kubectl).

On top of this, since release 18 of terraform-aws-modules/terraform-aws-eks, aws-auth isn't managed by the module anymore, most of the workarounds are based on exec/kubectl which is not something everyone can do.

[aidan-melen]

Here is the workaround we came up with, this will run as soon as the cluster is created in our case:

resource "kubernetes_job" "kube_system_labeler" {
  metadata {
    name      = "kube-system-labeler"
    namespace = utility
  }
  spec {
    template {
      metadata {}
      spec {
        service_account_name = utility
        container {
          name    = "ns-labeler"
          image   = "bitnami/kubectl:latest"
          command = ["/bin/sh", "-c", "kubectl label ns kube-system --overwrite config.linkerd.io/admission-webhooks=disabled admission.gatekeeper.sh/ignore=no-self-managing"]
        }
        restart_policy = "Never"
      }
    }
  }
  wait_for_completion = true
  timeouts {
    create = "5m"
  }
}

Can you please describe how this is a workaround?

[robertb724]

What we are trying to solve is managing the resources managed by the cloud provider. In my case, GKE itself manages the kube-system namespace. We are installing linkerd, which requires that the kube-system namespace has the label config.linkerd.io/admission-webhooks=disabled. We can not manage the kube-system as a tf resource since it would collide with what gke has created. This spins up a pod as soon as the cluster is ready(since it is using the k8s provider backed by this cluster) to label the namespace for us. It will fail if the label is not applied and we can control when it is executed, just as if we were able to interact with the namespace directly from the terraform execution.

[Speculor]

I am also hitting this problem because I must annotate the FluxCD.io kustomize-controller serviceaccount (which was created with the Terraform flux provider) with the name of the IAM role that allows for the KMS key used to encrypt k8s secrets to be used by Flux, ie:

kubectl -n flux-system annotate serviceaccount kustomize-controller --field-manager=flux-client-side-apply eks.amazonaws.com/role-arn='arn:aws:iam::<ACCOUNT_ID>:role/<KMS-ROLE-NAME>'

References: https://fluxcd.io/docs/guides/mozilla-sops/ https://registry.terraform.io/providers/fluxcd/flux/latest/docs/guides/github

We really need a kubernetes_patch resource, this is long overdue

[fabienpelletier]

I'm also hitting this issue when deploying tigera-operator on a new cluster. Enabling the Wireguard encryption[0] requires me to do kubectl patch felixconfiguration default --type='merge' -p '{"spec":{"wireguardEnabled":true}}' which I am currently running in a null_resource's local-provisioner.

[0] https://projectcalico.docs.tigera.io/security/encrypt-cluster-pod-traffic

[aidanmelen]

A second approach would be a new resource like kubernetes_manifest_field that manages only one field of an existing resource. That could be the more terraform way.

This would be a nice "terraformic" solution for an ugly problem caused by terraform needing to partially manage objects.