Closed ysineil closed 6 months ago
It generally seems to boil down to any changes don't get reflected correctly or complete. I deployed a new cluster via TF (using the same config above) and triggered the ugprade via TMC and it upgraded successfully.
@Axpz - Can you take up this issue?
It generally seems to boil down to any changes don't get reflected correctly or complete. Cluster Status: UPGRADING, Cluster Health: HEALTHY
Normally, this is supported. From the log, it seems like this is a backend issue, not a terraform issue.
I'm not sure it's a TMC/backend issue - if I create a cluster with TF and manually upgrade it via TMC (clicking the upgrade button) it works. Just when triggered via TF it goes in to an odd state.
Hi @ysineil @ramya-bangera, finally I can reproduce this issue. After terraform apply
on the version update from "v1.23.8+vmware.2-tkg.2-zshippable" -> "v1.24.9+vmware.1-tkg.4", I found below error message with kubectl describe cluster tfu-100-1 -n testns
Message: error computing the desired state of the Cluster topology: failed to apply patches: failed to generate patches for patch "nodeLabels": failed to generate JSON patches for item with uid "34a77e17-4945-4e10-9dd8-b1f4a93177bc": failed to calculate value for template: failed to render template: "run.tanzu.vmware.com/tkr={{ index (index .TKR_DATA .builtin.machineDeployment.version).labels \"run.tanzu.vmware.com/tkr\" }},run.tanzu.vmware.com/kubernetesDistributionVersion={{ index (index .TKR_DATA .builtin.machineDeployment.version).labels \"run.tanzu.vmware.com/tkr\" }},{{- range .nodePoolLabels }}{{ .key }}={{ .value }},{{- end }}\n": template: tpl:1:28: executing "tpl" at <index (index .TKR_DATA .builtin.machineDeployment.version).labels "run.tanzu.vmware.com/tkr">: error calling index: index of untyped nil
I think that's the root cause, but I still need some time to investigate how to fix it, I mean whether the terraform provider to fix it or upstream(TMC even TKG).
Thanks @Axpz - unsure if related but when doing a terraform import on a workload cluster we do get the TKR_DATA as part of the tfstate but not on a new cluster deployed via TF.
Example import: "cluster_variables": "{\"TKR_DATA\":{\"v1.23.8+vmware.2\":{\"kubernetesSpec\":{\"coredns\":{\"imageTag\":\"v1.8.6_vmware.7\"},\"etcd\":{\"imageTag\":\"v3.5.4_vmware.6\"},\"imageRepository\":\"localhost:5000/vmware.io\",\"version\":\"v1.23.8+vmware.2\"},\"labels\":{\"image-type\":\"vmi\",\"os-arch\":\"amd64\",\"os-name\":\"photon\",\"os-type\":\"linux\",\"os-version\":\"3\",\"run.tanzu.vmware.com/os-image\":\"ob-20611023-photon-3-amd64-vmi-k8s-v1.23.8---vmware.2-tkg.1-zshippable\",\"run.tanzu.vmware.com/tkr\":\"v1.23.8---vmware.2-tkg.2-zshippable\",\"vmi-name\":\"ob-20611023-photon-3-amd64-vmi-k8s-v1.23.8---vmware.2-tkg.1-zsh\"},\"osImageRef\":{\"name\":\"ob-20611023-photon-3-amd64-vmi-k8s-v1.23.8---vmware.2-tkg.1-zshippable\"}}},\"clusterEncryptionConfigYaml\":\"removed",\"extensionCert\":{\"contentSecret\":{\"key\":\"tls.crt\",\"name\":\"pcb-prod-nsxintel-extensions-ca\"}},\"nodePoolVolumes\":[{\"capacity\":{\"storage\":\"200G\"},\"mountPath\":\"/var/lib/containerd\",\"name\":\"containerd-0\",\"storageClass\":\"pcb-ha-zone-1-pv\"}],\"ntp\":\"0.0.0.0\",\"storageClass\":\"pcb-ha-zone-1-pv\",\"user\":{\"passwordSecret\":{\"key\":\"ssh-passwordkey\",\"name\":\"removed"},\"sshAuthorizedKey\":\"removed"},\"vmClass\":\"best-effort-medium\"}",
example new cluster: "cluster_variables": "{\"defaultStorageClass\":\"pcb-ha-zone-1-pv\",\"nodePoolVolumes\":[{\"capacity\":{\"storage\":\"200G\"},\"mountPath\":\"/var/lib/containerd\",\"name\":\"containerd\",\"storageClass\":\"pcb-ha-zone-1-pv\"}],\"storageClass\":\"pcb-ha-zone-1-pv\",\"storageClasses\":[\"pcb-ha-zone-1-pv\"],\"vmClass\":\"best-effort-medium\"}",
Hi @ysineil, this issue will be fix together with 364, hoping this can be done in next week considering the test and promotion.
Hi @ysineil, this issue have been fixed and promoted into PROD stage from TMC side, sorry for some delay because of tech issue. You can take a retry and if you have any questions just feel free to let us know, thanks!
Closing this issue as per https://github.com/vmware/terraform-provider-tanzu-mission-control/issues/364#issuecomment-1928190241
Describe the bug
Upgrading workload clusters from v1.23.8+vmware.2-tkg.2-zshippable to v1.24.9+vmware.1-tkg.4 by adjusting spec.version does not function properly.
Terraform will perform the following actions:
Relavent main.tf
vars:
Reproduction steps
Expected behavior
Upgrade to complete successfully.
Additional context
No response