kbst / terraform-provider-kustomization

Terraform provider for Kustomize
https://www.kubestack.com
Apache License 2.0
265 stars 53 forks source link

Cannot apply kustomization overlay #230

Closed td4b closed 1 year ago

td4b commented 1 year ago

I cant currently apply my overlay properly, seems like something chaged between the provider versions.

terraform apply
╷
│ Error: Invalid for_each argument
│ 
│   on kustomize.tf line 82, in resource "kustomization_resource" "vault_kst_resource":
│   82:   for_each = data.kustomization_overlay.kust_vault.ids
│     ├────────────────
│     │ data.kustomization_overlay.kust_vault.ids is a set of string, known only after apply
│ 
│ The "for_each" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be
│ created. To work around this, use the -target argument to first apply only the resources that the for_each depends on.

This makes no sense since this is a data resource defined with HCL it should have no dependecy here.

td4b commented 1 year ago

I have a simple patch

data "kustomization_overlay" "kust_vault" {

  resources = [
    "kustomize/"
  ]

  patches {
    target {
      kind = "ServiceAccount"
      name = "vault"
      namespace = "vault"
      annotation_selector = "eks.amazonaws.com/role-arn"
    }
    patch = <<-EOF
    - op: add
      path: "/metadata/annotations"
      value:
        eks.amazonaws.com/role-arn: "${module.iam_assumable_role_vault.iam_role_arn}"
    EOF
  }
}

that cant be applied via:

resource "kustomization_resource" "vault_kst_resource" {
  for_each = data.kustomization_overlay.kust_vault.ids
  manifest = data.kustomization_overlay.kust_vault.manifests[each.value]
}
td4b commented 1 year ago

I managed to get by this issue by applying the data resource first and then trying to apply the kustomization resource, however now I get another error:

github.com/kbst/terraform-provider-kustomize/kustomize.kustomizationResourceDiff: "policy/PodDisruptionBudget/vault/vault": api error: no matches for kind "PodDisruptionBudget" in version "policy/v1beta1"
td4b commented 1 year ago

looks like I need to downgrade my cluster to support this, FYI its not great that you have to manually apply the data resource before using the customize resource properly, as well as that depends_on for whatever reason is not respected in the kustomize resource in cases where an overlay is used..

pst commented 1 year ago

Data sources that depend on known after apply values are pushed into the apply phase, for_each doesn't work with that. This is common to Terraform and not specific to this provider. It's just that using this provider almost always also uses for_each. But it doesn't have to. If you know the resources and they're not changing frequently, you can do without for_each on the ids and instead have one kustomization_resource hardcoded for each resource in the kustomization. Like you would with any other provider. It's just a pita if you consume upstream YAML.