kubernetes_env does not allow applying environment variable to a daemonset.

[chrismaes87]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: 1.7.3
Kubernetes provider version: 2.25.2
Kubernetes version: 1.27

Affected Resource(s)

• kubernetes_env

Terraform Configuration Files

locals {
  containers = {
    "aws-node" = {
      container      = "aws-node"
      init_container = null
    }
    "aws-eks-nodeagent" = {
      container      = "aws-eks-nodeagent"
      init_container = null
    }
    "aws-vpc-cni-init" = {
      container      = null
      init_container = "aws-vpc-cni-init"
    }
  }

  envs = [
    {
      name  = "AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG"
      value = true
    }
  ]
}

resource "kubernetes_env" "aws_node-env" {
  for_each       = local.containers
  kind           = "DaemonSet"
  container      = each.value.container
  init_container = each.value.init_container
  metadata {
    name      = "aws-node"
    namespace = "kube-system"
  }

  api_version = "apps/v1"
  force       = true

  dynamic "env" {
    for_each = local.envs
    content {
      name  = env.value["name"]
      value = env.value["value"]
    }
  }
}

Debug Output

Panic Output

Steps to Reproduce

1. terraform apply works fine without error

Expected Behavior

kubectl -n kube-system get daemonset aws-node -o yaml | grep -C 1 AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG

should give something like (value true three times)

          value: "9001"
        - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
          value: "true"
--
              fieldPath: spec.nodeName
        - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
          value: "true"
--
          value: "false"
        - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
          value: "true"

Actual Behavior

the value is never true on all three resources. I will get something like

kubectl -n kube-system get daemonset aws-node -o yaml | grep -C 1 AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG

          value: "9001"
        - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
          value: "true"
--
              fieldPath: spec.nodeName
        - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
        - name: ENI_CONFIG_LABEL_DEF
--
          value: "false"
        - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
        - name: ENI_CONFIG_LABEL_DEF

it seems like kubernetes_env applies the environment variable to one container, and it gets removed from the others. I want the environment variable to be set on all containers of the daemonset.

Important Factoids

The problem is not linked to me parametrizing the three kubernetes_env resources. I have tested with three separate resources with everything hardcoded and this gave the exact same behaviour.

References

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[sheneska]

Hi @chrismaes87, you can try setting the field_manager to being the containers' value, for example field_manager = each.value.container. While this may work for this specific issue, there are some limitations here that would cause it not to work in every case. Please keep in mind that modifying resources that are outside of terraform is not encouraged and should be used only as a last resort.

[chrismaes87]

I tried this, and also set force=true but to no avail. He is managing to change the field value in some way, but not for all containers at once.

[joewragg]

I'm also seeing the same issue. Can we get fix for this it appears that either container or init_container are mandatory