search

gavinbunney / terraform-provider-kubectl

Terraform provider to handle raw kubernetes manifest yaml files
https://registry.terraform.io/providers/gavinbunney/kubectl
Mozilla Public License 2.0
609 stars 102 forks source link

resource [v1/Secret] isn't valid for cluster #288

Closed dariusj1 closed 5 months ago

dariusj1 commented 5 months ago

I'm sorry, I must be doing something wrong, but I just can't figure out what it is.

I have a terraform project creating an EKS and setting up kubectl locally and I also want to deploy several essential resources to the cluster using manifests.

resource "kubectl_manifest" "aws_secrets_key" {
  depends_on = [module.eks]

    yaml_body = <<-EOF
      apiVersion: v1
      kind: Secret
      metadata:
        namespace: external-secrets
        name: awssm-secret
      data:
        access-key: CHANGEME
        secret-access-key: CHANGEME
    EOF
}

terraform plan generates this

  # kubectl_manifest.aws_secrets_key will be created
  + resource "kubectl_manifest" "aws_secrets_key" {
      + api_version             = "v1"
      + apply_only              = false
      + force_conflicts         = false
      + force_new               = false
      + id                      = (known after apply)
      + kind                    = "Secret"
      + live_manifest_incluster = (sensitive value)
      + live_uid                = (known after apply)
      + name                    = "awssm-secret"
      + namespace               = "external-secrets"
      + server_side_apply       = false
      + uid                     = (known after apply)
      + validate_schema         = true
      + wait_for_rollout        = true
      + yaml_body               = (sensitive value)
      + yaml_body_parsed        = <<-EOT
            apiVersion: v1
            data: (sensitive value)
            kind: Secret
            metadata:
              name: awssm-secret
              namespace: external-secrets
        EOT
      + yaml_incluster          = (sensitive value)
    }

and terraform apply fails with this

│ Error: external-secrets/awssm-secret failed to create kubernetes rest client for update of resource: resource [v1/Secret] isn't valid for cluster, check the APIVersion and Kind fields are valid
│ 
│   with kubectl_manifest.aws_secrets_key,
│   on 13_k8s_tools.tf line 20, in resource "kubectl_manifest" "aws_secrets_key":
│   20: resource "kubectl_manifest" "aws_secrets_key" {
│

I also tried TF_LOG=DBUG and got this

2024-03-19T12:23:12.221+0200 [DEBUG] provider.terraform-provider-aws_v5.41.0_x5: [DEBUG] Waiting for state to become: [available storage-optimization]
2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: 2024/03/19 12:23:14 [DEBUG] external-secrets/awssm-secret Unstructed YAML: map[apiVersion:v1 data:map[access-key:CHANGEME secret-access-key:CHANGEME] kind:Secret metadata:map[name:awssm-secret namespace:external-secrets]]
2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: 2024/03/19 12:23:14 [DEBUG] external-secrets/awssm-secret apply kubernetes resource:
2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: apiVersion: v1
2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: kind: Secret
2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: metadata:
2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0:   namespace: external-secrets
2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0:   name: awssm-secret
2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: data:
2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0:   access-key: CHANGEME
2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0:   secret-access-key: CHANGEME
2024-03-19T12:23:14.510+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: 2024/03/19 12:23:14 [ERROR] creating manifest failed: external-secrets/awssm-secret failed to create kubernetes rest client for update of resource: resource [v1/Secret] isn't valid for cluster, check the APIVersion and Kind fields are valid

now, if I kubectl apply -f- that same manifest, the secret is created successfully.

What am I doing wrong?

alekc commented 5 months ago

One is using a provider which is not being updated (I would humbly suggest https://registry.terraform.io/providers/alekc/kubectl/2.0.4)

Second, imho if you are dealing with standard objects kubernetes provider is much stable https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/secret.html

On Tue, 19 Mar 2024, 11:50 dariusj, @.***> wrote:

I'm sorry, I must be doing something wrong, but I just can't figure out what it is.

I have a terraform project creating an EKS and setting up kubectl locally and I also want to deploy several essential resources to the cluster using manifests.

resource "kubectl_manifest" "aws_secrets_key" { depends_on = [module.eks]

yaml_body = <<-EOF      apiVersion: v1      kind: Secret      metadata:        namespace: external-secrets        name: awssm-secret      data:        access-key: CHANGEME        secret-access-key: CHANGEME    EOF

}

terraform plan generates this

kubectl_manifest.aws_secrets_key will be created

  • resource "kubectl_manifest" "aws_secrets_key" {
    • api_version = "v1"
    • apply_only = false
    • force_conflicts = false
    • force_new = false
    • id = (known after apply)
    • kind = "Secret"
    • live_manifest_incluster = (sensitive value)
    • live_uid = (known after apply)
    • name = "awssm-secret"
    • namespace = "external-secrets"
    • server_side_apply = false
    • uid = (known after apply)
    • validate_schema = true
    • wait_for_rollout = true
    • yaml_body = (sensitive value)
    • yaml_body_parsed = <<-EOT apiVersion: v1 data: (sensitive value) kind: Secret metadata: name: awssm-secret namespace: external-secrets EOT
    • yaml_incluster = (sensitive value) }

and terraform apply fails with this

│ Error: external-secrets/awssm-secret failed to create kubernetes rest client for update of resource: resource [v1/Secret] isn't valid for cluster, check the APIVersion and Kind fields are valid │ │ with kubectl_manifest.aws_secrets_key, │ on 13_k8s_tools.tf line 20, in resource "kubectl_manifest" "aws_secrets_key": │ 20: resource "kubectl_manifest" "aws_secrets_key" { │

I also tried TF_LOG=DBUG and got this

2024-03-19T12:23:12.221+0200 [DEBUG] provider.terraform-provider-aws_v5.41.0_x5: [DEBUG] Waiting for state to become: [available storage-optimization] 2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: 2024/03/19 12:23:14 [DEBUG] external-secrets/awssm-secret Unstructed YAML: map[apiVersion:v1 data:map[access-key:CHANGEME secret-access-key:CHANGEME] kind:Secret metadata:map[name:awssm-secret namespace:external-secrets]] 2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: 2024/03/19 12:23:14 [DEBUG] external-secrets/awssm-secret apply kubernetes resource: 2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: apiVersion: v1 2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: kind: Secret 2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: metadata: 2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: namespace: external-secrets 2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: name: awssm-secret 2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: data: 2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: access-key: CHANGEME 2024-03-19T12:23:14.488+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: secret-access-key: CHANGEME 2024-03-19T12:23:14.510+0200 [DEBUG] provider.terraform-provider-kubectl_v1.14.0: 2024/03/19 12:23:14 [ERROR] creating manifest failed: external-secrets/awssm-secret failed to create kubernetes rest client for update of resource: resource [v1/Secret] isn't valid for cluster, check the APIVersion and Kind fields are valid

now, if I kubectl apply -f- that same manifest, the secret is created successfully.

What am I doing wrong?

— Reply to this email directly, view it on GitHub https://github.com/gavinbunney/terraform-provider-kubectl/issues/288, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACJ5V5R6RPACENLNJNWVRTYZAJ5ZAVCNFSM6AAAAABE5JQQYOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGE4TINRSHA2TIMY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

dariusj1 commented 5 months ago

@alekc

using a provider which is not being updated

yeah, that's right. I used the version from README.md rather than from the registry. In registry the newest version seems to be 1.14.0

2.0.4 is in another repository which seems to be forked off the Gavin's kubectl. I'll give it a spin, thanks

kubernetes provider is much stable

perhaps, but it doesn't deal well with manifests involving CRDs. Which is the only reason why I picked the kubectl provider https://github.com/hashicorp/terraform-provider-kubernetes/issues/1367 https://github.com/hashicorp/terraform-provider-kubernetes/issues/1583

dariusj1 commented 5 months ago

Upgrading the version worked. Thank you.

alekc commented 5 months ago

@dariusj1 thats' why I said "imho if you are dealing with standard objects". As long as it's standard (secrets, ns, deployments, etc), you will get a better time especially with diffs if using kubernetes provider and kubectl for non standard manifests.