search

hashicorp / terraform-provider-kubernetes-alpha

A Terraform provider for Kubernetes that uses dynamic resource types and server-side apply. Supports all Kubernetes resources.
https://registry.terraform.io/providers/hashicorp/kubernetes-alpha/latest
Mozilla Public License 2.0
490 stars 63 forks source link

CRD dependency error when CR definition and CR instance in same TF project #247

Open tringuyen-yw opened 3 years ago

tringuyen-yw commented 3 years ago

Terraform, Provider, Kubernetes versions

Terraform v1.0.2
on linux_amd64
+ provider registry.terraform.io/hashicorp/kubernetes-alpha v0.5.0

Kubernetes version: 1.19.11

Affected Resource(s)

Terraform Configuration Files

terraform {
  required_version = ">= 1.0.0"

  required_providers {
    kubernetes-alpha = {
      source = "hashicorp/kubernetes-alpha"
      version = "0.5.0"
    }
  }
}

provider "kubernetes-alpha" {
  config_path = "~/.kube/config"
  config_context = "my-sandbox-context"
}

resource "kubernetes_manifest" "trillu_crd" {
  provider = kubernetes-alpha

  # see attached kubernetes-alpha_CRD-dependency-error.zip for the yaml content ("Actual Behavior" section)
  manifest = yamldecode(file("./Trillucination_CRD.yaml"))
}

resource "kubernetes_manifest" "trillu_instance" {
  provider = kubernetes-alpha

  # see attached kubernetes-alpha_CRD-dependency-error.zip for the yaml content ("Actual Behavior" section)
  manifest = yamldecode(file("./Trillucination_EagleNebula.yaml"))

  depends_on = [
    kubernetes_manifest.trillu_crd
  ]
}

Error

terraform plan
╷
│ Error: Failed to determine GroupVersionResource for manifest
│ 
│   with kubernetes_manifest.trillu_instance,
│   on main.tf line 24, in resource "kubernetes_manifest" "trillu_instance":
│   24: resource "kubernetes_manifest" "trillu_instance" {
│ 
│ no matches for kind "Trillucination" in group "tridebug.edu"

Expected Behavior

The kubernetes-alpha provider should honor the depends_on clause. ie waiting for the CR definition to exist before checking on the state of the CR object instance.

Actual Behavior

There are similar issues filed related to Error: Failed to determine GroupVersionResource for manifest. The issue I am filing here is minimal and generic, using a self-made CRD which doesn't depend on any helm chart or other resources. Also use latest version as of 2021-07-09.

TERRAFORM CONFIG + YAML resources: kubernetes-alpha_CRD-dependency-error.zip

I guess the cause might be that kubernetes-alpha uses Server Side Apply (SSA). As stated in the blog Deploy Any Resource With The New Kubernetes Provider for HashiCorp Terraform

With SSA enabled, the provider defers to the Kubernetes API to detect differences in attributes and handle conflict detection for Terraform plan and apply

Because kubernetes-alpha attemps to query the Kubernetes API about the state of the CR instance, in my example, the Trillucination doesn't exist yet on the server side, causing terraform plan to fail. Unfortunately, that CR could not exist yet, because it is the job of the TF config to create it ftrst.

When it comes to query Kunermetes API about Custom Resource, there is a dependency chain: CR definition must exist before CR object instance could be created. We solve that manually by sequencing the kubectl apply -f ... in 2 separate steps. But terraform plan tries to make the inventory of it's state in one go and find itself in a kind of "resource not found" situation.

I think the kubernetes-alpha should handle this "CR dependency" chain by skipping the check on the dependant CR instances when the CR definition itself is created within the same TF config/project. And just consider the CR instance as not yet exists.

Important Factoids

References

Community Note