search

vmware / terraform-provider-tanzu-mission-control

Terraform provider to manage resources of Tanzu Mission Control
Mozilla Public License 2.0
38 stars 31 forks source link

Issue modifying spec.cluster_group_name #364

Closed ysineil closed 6 months ago

ysineil commented 7 months ago

Describe the bug

When adjusting spec.cluster_group_name from one group to another, the action times out and does not complete. In our case, we had hard coded cluster_group_name = "default" and when making it more dynamic, it failed. New cluster creation with the same group definition works.

main.tf section:

resource "tanzu-mission-control_tanzu_kubernetes_cluster" "tkgs_cluster" {
  management_cluster_name = var.management_cluster
  provisioner_name = var.provisioner
  name = var.name

  spec {
    cluster_group_name = "${var.location}"

    topology {
      version = replace("${var.k8s_version}", "+", "+")
      cluster_class = "tanzukubernetescluster"
      cluster_variables = jsonencode(local.tkgs_cluster_variables)

      control_plane {
        replicas = 3

        os_image {
          name = "${var.os_image}"
          version = "3"
          arch = "amd64"
        }
      }

      nodepool {
        name = "default-nodepool-a"
        description = "tkgs workload nodepool"

        spec {
          worker_class = "node-pool"
          replicas = 3
          overrides = jsonencode(local.tkgs_cluster_variables)
          failure_domain = "${var.location}-zone-a"

          os_image {
            name = "${var.os_image}"
            version = "3"
            arch = "amd64"
          }
        }
      }

      nodepool {
        name = "default-nodepool-b"
        description = "tkgs workload nodepool"

        spec {
          worker_class = "node-pool"
          replicas = 3
          overrides = jsonencode(local.tkgs_cluster_variables)
          failure_domain = "${var.location}-zone-b"

          os_image {
            name = "${var.os_image}"
            version = "3"
            arch    = "amd64"
          }
        }
      }

      nodepool {
        name        = "default-nodepool-c"
        description = "tkgs workload nodepool"

        spec {
          worker_class   = "node-pool"
          replicas       = 3
          overrides      = jsonencode(local.tkgs_cluster_variables)
          failure_domain = "${var.location}-zone-c"

          os_image {
            name    = "${var.os_image}"
            version = "3"
            arch    = "amd64"
          }
        }
      }   

      network {
        pod_cidr_blocks = [
          "172.20.0.0/16",
        ]
        service_cidr_blocks = [
          "10.96.0.0/16",
        ]
        service_domain = "cluster.local"
      }
    }
  }

  timeout_policy {
    timeout             = 60
    wait_for_kubeconfig = true
    fail_on_timeout     = true
  }
}`

Reproduction steps

  1. Adjust spec.cluster_group_name and apply change.

Expected behavior

Cluster group to update

Additional context

No response

Axpz commented 7 months ago

Hi @ysineil, @ramya-bangera, I cann't reproduce this issue, please let me know if there is something I missed, and please find my reproduce steps (please ignore the 1 added, I was doing another test simultaneous but w/o affect):

  1. Created a cluster and update the cluster_group_name

    
# tanzu-mission-control_tanzu_kubernetes_cluster.tkgs_cluster will be updated in-place
~ resource "tanzu-mission-control_tanzu_kubernetes_cluster" "tkgs_cluster" {
    id                      = "tf-mc/testns/tfu-100"
    name                    = "tfu-100"
    # (2 unchanged attributes hidden)

  ~ spec {
      ~ cluster_group_name = "default" -> "demo-cluster-group"
        # (2 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

    # (2 unchanged blocks hidden)
}

Plan: 1 to add, 1 to change, 0 to destroy.


2. After Entered yes, succeeded with terraform apply

Apply complete! Resources: 1 added, 1 changed, 0 destroyed.

3. Started another terraform plan, and get output

terraform plan
data.tanzu-mission-control_cluster_class.tkc-cc: Reading... tanzu-mission-control_tanzu_kubernetes_cluster.tkgs_cluster: Refreshing state... [id=tf-mc/testns/tfu-100] data.tanzu-mission-control_cluster_class.tkc-cc: Read complete after 3s [id=tf-mc/testns/tanzukubernetescluster] local_file.foo: Refreshing state... [id=618be9bcf5d1abd28679479ecf8d535be5acc655]

No changes. Your infrastructure matches the configuration.


4. Double check from the TMC, found that the cluster group is already changed to `demo-cluster-group`
<img width="1225" alt="image" src="https://github.com/vmware/terraform-provider-tanzu-mission-control/assets/68174603/23d2c538-6502-41a7-a320-56e545ab02c7">
ysineil commented 7 months ago

Interesting - I get the following when checking the state of the cluster via kubectl:

status: conditions:

Possible to share the output of your state file?

Axpz commented 7 months ago

Forget to mention, I am reproducing this issue with single control_plane, let me start a test with high-available mode. And this is my state file

{
  "version": 4,
  "terraform_version": "1.5.2",
  "serial": 7,
  "lineage": "f343d17f-0be0-31f8-8416-8fd300f5206f",
  "outputs": {
    "cluster_class_variables_schema": {
      "value": "{\"TKR_DATA\":{\"additionalProperties\":{\"properties\":{\"kubernetesSpec\":{\"properties\":{\"coredns\":{\"properties\":{\"imageTag\":{\"type\":\"string\"}},\"type\":\"object\"},\"etcd\":{\"properties\":{\"imageTag\":{\"type\":\"string\"}},\"type\":\"object\"},\"imageRepository\":{\"type\":\"string\"},\"version\":{\"type\":\"string\"}},\"type\":\"object\"},\"labels\":{\"additionalProperties\":{\"type\":\"string\"},\"type\":\"object\"},\"osImageRef\":{\"properties\":{\"name\":{\"type\":\"string\"}},\"type\":\"object\"}},\"type\":\"object\"},\"type\":\"object\"},\"clusterEncryptionConfigYaml\":{\"type\":\"string\"},\"controlPlaneCertificateRotation\":{\"default\":{\"activate\":false},\"properties\":{\"activate\":{\"default\":true,\"type\":\"boolean\"},\"daysBefore\":{\"default\":90,\"format\":\"int32\",\"minimum\":7,\"type\":\"integer\"}},\"type\":\"object\"},\"controlPlaneVolumes\":{\"items\":{\"properties\":{\"capacity\":{\"properties\":{\"storage\":{\"type\":\"string\"}},\"type\":\"object\"},\"mountPath\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"},\"storageClass\":{\"type\":\"string\"}},\"type\":\"object\"},\"type\":\"array\"},\"defaultRegistrySecret\":{\"properties\":{\"data\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"},\"namespace\":{\"type\":\"string\"}},\"type\":\"object\"},\"defaultStorageClass\":{\"type\":\"string\"},\"defaultVolumeSnapshotClass\":{\"type\":\"string\"},\"extensionCert\":{\"properties\":{\"contentSecret\":{\"properties\":{\"key\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"}},\"type\":\"object\"}},\"type\":\"object\"},\"nodePoolLabels\":{\"items\":{\"properties\":{\"key\":{\"type\":\"string\"},\"value\":{\"type\":\"string\"}},\"type\":\"object\"},\"type\":\"array\"},\"nodePoolTaints\":{\"items\":{\"properties\":{\"effect\":{\"type\":\"string\"},\"key\":{\"type\":\"string\"},\"timeAdded\":{\"type\":\"integer\"},\"value\":{\"type\":\"string\"}},\"type\":\"object\"},\"type\":\"array\"},\"nodePoolVolumes\":{\"items\":{\"properties\":{\"capacity\":{\"properties\":{\"storage\":{\"type\":\"string\"}},\"type\":\"object\"},\"mountPath\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"},\"storageClass\":{\"type\":\"string\"}},\"type\":\"object\"},\"type\":\"array\"},\"ntp\":{\"type\":\"string\"},\"proxy\":{\"properties\":{\"httpProxy\":{\"type\":\"string\"},\"httpsProxy\":{\"type\":\"string\"},\"noProxy\":{\"items\":{\"type\":\"string\"},\"type\":\"array\"}},\"type\":\"object\"},\"storageClass\":{\"required\":true,\"type\":\"string\"},\"storageClasses\":{\"items\":{\"type\":\"string\"},\"type\":\"array\"},\"trust\":{\"properties\":{\"additionalTrustedCAs\":{\"items\":{\"properties\":{\"name\":{\"type\":\"string\"}},\"type\":\"object\"},\"type\":\"array\"}},\"type\":\"object\"},\"user\":{\"properties\":{\"passwordSecret\":{\"properties\":{\"key\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"}},\"type\":\"object\"},\"sshAuthorizedKey\":{\"type\":\"string\"}},\"type\":\"object\"},\"vmClass\":{\"required\":true,\"type\":\"string\"},\"volumeSnapshotClasses\":{\"items\":{\"type\":\"string\"},\"type\":\"array\"}}",
      "type": "string"
    },
    "cluster_class_variables_template": {
      "value": "{\"TKR_DATA\":{\"custom_key\":{\"kubernetesSpec\":{\"coredns\":{\"imageTag\":\"String\"},\"etcd\":{\"imageTag\":\"String\"},\"imageRepository\":\"String\",\"version\":\"String\"},\"labels\":{\"custom_key\":\"String\"},\"osImageRef\":{\"name\":\"String\"}}},\"clusterEncryptionConfigYaml\":\"String\",\"controlPlaneCertificateRotation\":{\"activate\":false,\"daysBefore\":7},\"controlPlaneVolumes\":[{\"capacity\":{\"storage\":\"String\"},\"mountPath\":\"String\",\"name\":\"String\",\"storageClass\":\"String\"}],\"defaultRegistrySecret\":{\"data\":\"String\",\"name\":\"String\",\"namespace\":\"String\"},\"defaultStorageClass\":\"String\",\"defaultVolumeSnapshotClass\":\"String\",\"extensionCert\":{\"contentSecret\":{\"key\":\"String\",\"name\":\"String\"}},\"nodePoolLabels\":[{\"key\":\"String\",\"value\":\"String\"}],\"nodePoolTaints\":[{\"effect\":\"String\",\"key\":\"String\",\"timeAdded\":1,\"value\":\"String\"}],\"nodePoolVolumes\":[{\"capacity\":{\"storage\":\"String\"},\"mountPath\":\"String\",\"name\":\"String\",\"storageClass\":\"String\"}],\"ntp\":\"String\",\"proxy\":{\"httpProxy\":\"String\",\"httpsProxy\":\"String\",\"noProxy\":[\"String\"]},\"storageClass\":\"String\",\"storageClasses\":[\"String\"],\"trust\":{\"additionalTrustedCAs\":[{\"name\":\"String\"}]},\"user\":{\"passwordSecret\":{\"key\":\"String\",\"name\":\"String\"},\"sshAuthorizedKey\":\"String\"},\"vmClass\":\"String\",\"volumeSnapshotClasses\":[\"String\"]}",
      "type": "string"
    }
  },
  "resources": [
    {
      "mode": "data",
      "type": "tanzu-mission-control_cluster_class",
      "name": "tkc-cc",
      "provider": "provider[\"vmware/dev/tanzu-mission-control\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "id": "tf-mc/testns/tanzukubernetescluster",
            "management_cluster_name": "tf-mc",
            "name": "tanzukubernetescluster",
            "provisioner_name": "testns",
            "variables_schema": "{\"TKR_DATA\":{\"additionalProperties\":{\"properties\":{\"kubernetesSpec\":{\"properties\":{\"coredns\":{\"properties\":{\"imageTag\":{\"type\":\"string\"}},\"type\":\"object\"},\"etcd\":{\"properties\":{\"imageTag\":{\"type\":\"string\"}},\"type\":\"object\"},\"imageRepository\":{\"type\":\"string\"},\"version\":{\"type\":\"string\"}},\"type\":\"object\"},\"labels\":{\"additionalProperties\":{\"type\":\"string\"},\"type\":\"object\"},\"osImageRef\":{\"properties\":{\"name\":{\"type\":\"string\"}},\"type\":\"object\"}},\"type\":\"object\"},\"type\":\"object\"},\"clusterEncryptionConfigYaml\":{\"type\":\"string\"},\"controlPlaneCertificateRotation\":{\"default\":{\"activate\":false},\"properties\":{\"activate\":{\"default\":true,\"type\":\"boolean\"},\"daysBefore\":{\"default\":90,\"format\":\"int32\",\"minimum\":7,\"type\":\"integer\"}},\"type\":\"object\"},\"controlPlaneVolumes\":{\"items\":{\"properties\":{\"capacity\":{\"properties\":{\"storage\":{\"type\":\"string\"}},\"type\":\"object\"},\"mountPath\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"},\"storageClass\":{\"type\":\"string\"}},\"type\":\"object\"},\"type\":\"array\"},\"defaultRegistrySecret\":{\"properties\":{\"data\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"},\"namespace\":{\"type\":\"string\"}},\"type\":\"object\"},\"defaultStorageClass\":{\"type\":\"string\"},\"defaultVolumeSnapshotClass\":{\"type\":\"string\"},\"extensionCert\":{\"properties\":{\"contentSecret\":{\"properties\":{\"key\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"}},\"type\":\"object\"}},\"type\":\"object\"},\"nodePoolLabels\":{\"items\":{\"properties\":{\"key\":{\"type\":\"string\"},\"value\":{\"type\":\"string\"}},\"type\":\"object\"},\"type\":\"array\"},\"nodePoolTaints\":{\"items\":{\"properties\":{\"effect\":{\"type\":\"string\"},\"key\":{\"type\":\"string\"},\"timeAdded\":{\"type\":\"integer\"},\"value\":{\"type\":\"string\"}},\"type\":\"object\"},\"type\":\"array\"},\"nodePoolVolumes\":{\"items\":{\"properties\":{\"capacity\":{\"properties\":{\"storage\":{\"type\":\"string\"}},\"type\":\"object\"},\"mountPath\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"},\"storageClass\":{\"type\":\"string\"}},\"type\":\"object\"},\"type\":\"array\"},\"ntp\":{\"type\":\"string\"},\"proxy\":{\"properties\":{\"httpProxy\":{\"type\":\"string\"},\"httpsProxy\":{\"type\":\"string\"},\"noProxy\":{\"items\":{\"type\":\"string\"},\"type\":\"array\"}},\"type\":\"object\"},\"storageClass\":{\"required\":true,\"type\":\"string\"},\"storageClasses\":{\"items\":{\"type\":\"string\"},\"type\":\"array\"},\"trust\":{\"properties\":{\"additionalTrustedCAs\":{\"items\":{\"properties\":{\"name\":{\"type\":\"string\"}},\"type\":\"object\"},\"type\":\"array\"}},\"type\":\"object\"},\"user\":{\"properties\":{\"passwordSecret\":{\"properties\":{\"key\":{\"type\":\"string\"},\"name\":{\"type\":\"string\"}},\"type\":\"object\"},\"sshAuthorizedKey\":{\"type\":\"string\"}},\"type\":\"object\"},\"vmClass\":{\"required\":true,\"type\":\"string\"},\"volumeSnapshotClasses\":{\"items\":{\"type\":\"string\"},\"type\":\"array\"}}",
            "variables_template": "{\"TKR_DATA\":{\"custom_key\":{\"kubernetesSpec\":{\"coredns\":{\"imageTag\":\"String\"},\"etcd\":{\"imageTag\":\"String\"},\"imageRepository\":\"String\",\"version\":\"String\"},\"labels\":{\"custom_key\":\"String\"},\"osImageRef\":{\"name\":\"String\"}}},\"clusterEncryptionConfigYaml\":\"String\",\"controlPlaneCertificateRotation\":{\"activate\":false,\"daysBefore\":7},\"controlPlaneVolumes\":[{\"capacity\":{\"storage\":\"String\"},\"mountPath\":\"String\",\"name\":\"String\",\"storageClass\":\"String\"}],\"defaultRegistrySecret\":{\"data\":\"String\",\"name\":\"String\",\"namespace\":\"String\"},\"defaultStorageClass\":\"String\",\"defaultVolumeSnapshotClass\":\"String\",\"extensionCert\":{\"contentSecret\":{\"key\":\"String\",\"name\":\"String\"}},\"nodePoolLabels\":[{\"key\":\"String\",\"value\":\"String\"}],\"nodePoolTaints\":[{\"effect\":\"String\",\"key\":\"String\",\"timeAdded\":1,\"value\":\"String\"}],\"nodePoolVolumes\":[{\"capacity\":{\"storage\":\"String\"},\"mountPath\":\"String\",\"name\":\"String\",\"storageClass\":\"String\"}],\"ntp\":\"String\",\"proxy\":{\"httpProxy\":\"String\",\"httpsProxy\":\"String\",\"noProxy\":[\"String\"]},\"storageClass\":\"String\",\"storageClasses\":[\"String\"],\"trust\":{\"additionalTrustedCAs\":[{\"name\":\"String\"}]},\"user\":{\"passwordSecret\":{\"key\":\"String\",\"name\":\"String\"},\"sshAuthorizedKey\":\"String\"},\"vmClass\":\"String\",\"volumeSnapshotClasses\":[\"String\"]}",
            "worker_classes": [
              "node-pool"
            ]
          },
          "sensitive_attributes": []
        }
      ]
    },
    {
      "mode": "managed",
      "type": "tanzu-mission-control_tanzu_kubernetes_cluster",
      "name": "tkgs_cluster",
      "provider": "provider[\"vmware/dev/tanzu-mission-control\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "id": "tf-mc/testns/tfu-100",
            "management_cluster_name": "tf-mc",
            "meta": [
              {
                "annotations": {
                  "tmc-internal-uid": "c:01HKC2D5C4VE6WVN1D8D05TX54"
                },
                "description": "",
                "labels": {
                  "tmc.cloud.vmware.com/creator": "zxianping"
                },
                "resource_version": "873",
                "uid": "tkc:8c0930e9-e0fa-4b4b-ac5f-e13dcfc5883d"
              }
            ],
            "name": "tfu-100",
            "provisioner_name": "testns",
            "spec": [
              {
                "cluster_group_name": "demo-cluster-group",
                "image_registry": "",
                "kubeconfig": ".....",
                "proxy_name": "",
                "tmc_managed": true,
                "topology": [
                  {
                    "cluster_class": "tanzukubernetescluster",
                    "cluster_variables": "{\"controlPlaneCertificateRotation\":{\"activate\":true,\"daysBefore\":31},\"defaultStorageClass\":\"tkgs-k8s-obj-policy\",\"defaultVolumeSnapshotClass\":\"volumesnapshotclass-delete\",\"nodePoolLabels\":[],\"nodePoolVolumes\":[{\"capacity\":{\"storage\":\"20G\"},\"mountPath\":\"/var/lib/containerd\",\"name\":\"containerd\",\"storageClass\":\"tkgs-k8s-obj-policy\"},{\"capacity\":{\"storage\":\"20G\"},\"mountPath\":\"/var/lib/kubelet\",\"name\":\"kubelet\",\"storageClass\":\"tkgs-k8s-obj-policy\"}],\"storageClass\":\"tkgs-k8s-obj-policy\",\"storageClasses\":[\"tkgs-k8s-obj-policy\"],\"vmClass\":\"best-effort-medium\"}",
                    "control_plane": [
                      {
                        "meta": [],
                        "os_image": [
                          {
                            "arch": "amd64",
                            "name": "photon",
                            "version": "3"
                          }
                        ],
                        "replicas": 1
                      }
                    ],
                    "core_addon": [],
                    "network": [
                      {
                        "pod_cidr_blocks": [
                          "100.96.0.0/11"
                        ],
                        "service_cidr_blocks": [
                          "100.64.0.0/13"
                        ],
                        "service_domain": "cluster.local"
                      }
                    ],
                    "nodepool": [
                      {
                        "description": "simple small md",
                        "name": "md-0",
                        "spec": [
                          {
                            "failure_domain": "",
                            "meta": [],
                            "os_image": [
                              {
                                "arch": "amd64",
                                "name": "photon",
                                "version": "3"
                              }
                            ],
                            "overrides": "{\"storageClass\":\"tkgs-k8s-obj-policy\",\"vmClass\":\"best-effort-medium\"}",
                            "replicas": 1,
                            "worker_class": "node-pool"
                          }
                        ]
                      }
                    ],
                    "version": "v1.23.8+vmware.2-tkg.2-zshippable"
                  }
                ]
              }
            ],
            "timeout_policy": []
          },
          "sensitive_attributes": [],
          "private": "bnVsbA=="
        }
      ]
    }
  ],
  "check_results": null
}
Axpz commented 7 months ago

Hi @ysineil, we have been starting to fix this issue from backend (TMC UPDATE API), hoping this can be done in next week considering the test and promotion.

Axpz commented 6 months ago

Hi @ysineil, this issue have been fixed from TMC side, again sorry for some delay because of tech issue. You can take a retry and if you have any questions just feel free to let us know, thanks!

Axpz commented 6 months ago

Hi @ysineil, there is no change from terraform side, can I close this(as well as #363) if there is no other questions.

ysineil commented 6 months ago

changes look like they're working as expected now, thank you!

vmw-vjn commented 6 months ago

Thank you for confirming, closing issue.