tanzu kubernetes clusters upgrade

andrzej-dobrzynski-pushgaming commented 9 months ago

Hi @lvirbalas @adambarreiro

Description

We are upgrading tanzu kubernetes clusters based on the guide https://registry.terraform.io/providers/vmware/vcd/latest/docs/guides/container_service_extension_4_x_cluster_management#updating-a-kubernetes-cluster

... Upgradeable items: TKGm OVA: If there is a newer version of TKGm, we can modify the referenced OVA. ...

git diff

-input_entity_capiYaml_TKGVERSION             = "v2.2.0"
+input_entity_capiYaml_TKGVERSION             = "v2.4.0"
-input_entity_capiYaml_tanzuKubernetesRelease = "v1.25.7---vmware.2-tkg.1"
+input_entity_capiYaml_tanzuKubernetesRelease = "v1.27.7---vmware.2-tkg.1"

-input_entity_capiYaml_control_plane_node_pool_version         = "v1.25.7+vmware.2"
+input_entity_capiYaml_control_plane_node_pool_version         = "v1.27.7+vmware.2"
-input_entity_capiYaml_control_plane_node_pool_template        = "Ubuntu 20.04 and Kubernetes v1.25.7+vmware.2"
+input_entity_capiYaml_control_plane_node_pool_template        = "Ubuntu 20.04 and Kubernetes v1.27.7+vmware.2"

-input_entity_capiYaml_worker_node_pool_version         = "v1.25.7+vmware.2"
+input_entity_capiYaml_worker_node_pool_version         = "v1.27.7+vmware.2"
-input_entity_capiYaml_worker_node_pool_template        = "Ubuntu 20.04 and Kubernetes v1.25.7+vmware.2"
+input_entity_capiYaml_worker_node_pool_template        = "Ubuntu 20.04 and Kubernetes v1.27.7+vmware.2"

is this ok for the upgrade to complete ?

Affected Resource(s)

vcd_rde

Expected Behavior

upgrade is complete

Actual Behavior

tbc

Steps to Reproduce

terraform apply

adambarreiro commented 9 months ago

Hi @andrzej-dobrzynski-pushgaming,

As per CSE UI wizard, they only allow to upgrade to the immediate next Kubernetes minor, in this case, as per your snippet it should be Kubernetes v1.26.8 and TKG 2.3.1/2.4.0.

Apart from that, I think your snippet would be correct.

To sum up, these are the elements of the YAML that need to be upgraded:

In VCDMachineTemplate, the spec.template.spec.template: I believe this should be input_entity_capiYaml_control_plane_node_pool_template and input_entity_capiYaml_worker_node_pool_template in your snippet, references the new template name.
In MachineDeployment, the spec.template.spec.version: In your snippet probably it is input_entity_capiYaml_control_plane_node_pool_version and input_entity_capiYaml_worker_node_pool_version, it should be set to the new Kubernetes version, as you did.
In Cluster, metadata.annotations.TKGVERSION must be updated with the new TKG version. So input_entity_capiYaml_TKGVERSION looks good. Also metadata.labels.tanzuKubernetesRelease that corresponds to input_entity_capiYaml_tanzuKubernetesRelease
In KubeadmControlPlane, the spec.version should match with input_entity_capiYaml_control_plane_node_pool_version
Last, spec.kubeadmConfigSpec.clusterConfiguration.dns.imageTag and spec.kubeadmConfigSpec.clusterConfiguration.etcd.local.imageTag should be updated to the v1.26.8 versions. These can be picked from the v1.26.8 template: https://github.com/vmware/cluster-api-provider-cloud-director/blob/main/templates/cluster-template-v1.26.8-tkgv2.3.1-crs.yaml, dns.imageTag=v1.9.3_vmware.11 and etcd.local.imageTag=v3.5.6_vmware.14

To apply the YAML update, you will also need to take the most recent state of the cluster, which is reflected in the contents of the computed_entity attribute. You can use this output for that purpose:

output "computed_k8s_cluster" {
  value = vcd_rde.k8s_cluster_instance.computed_entity # References your cluster
}

And obtain the required data with the command:

terraform output -json computed_k8s_cluster > computed_cluster.json

Open the created computed_cluster.json file and copy the whole "status" object from the JSON. This status is vital to avoid losing information of the cluster on updates, so you need to put it back in the file that serves as RDE input in input_entity, which is tkgmcluster.json.template in the examples.

This file should look like this once updated:

{
  "apiVersion": "capvcd.vmware.com/v1.1",
  "kind": "CAPVCDCluster",
  "name": "${name}",
  "metadata": {
     ...omitted
  },
  "spec": {
    "vcdKe": {
      ...omitted
    },
    "capiYaml": ${capi_yaml}
  },
  "status": { ... } <-- The object from the 'computed_entity' attribute that we need
}

(Before applying an update, please verify that the cluster is in provisioned state, otherwise it can't be updated)

Once the YAML is patched, the CSE Server should eventually pick the changes and update the VMs/vApps.

andrzej-dobrzynski-pushgaming commented 9 months ago

Hi @adambarreiro

Thanks for quick reply

updated git diff

-input_entity_capiYaml_TKGVERSION             = "v2.2.0"
+input_entity_capiYaml_TKGVERSION             = "v2.3.1"
-input_entity_capiYaml_tanzuKubernetesRelease = "v1.25.7---vmware.2-tkg.1"
+input_entity_capiYaml_tanzuKubernetesRelease = "v1.26.8---vmware.2-tkg.1"

-input_entity_capiYaml_control_plane_node_pool_version         = "v1.25.7+vmware.2"
+input_entity_capiYaml_control_plane_node_pool_version         = "v1.26.8+vmware.2"
-input_entity_capiYaml_control_plane_node_pool_template        = "Ubuntu 20.04 and Kubernetes v1.25.7+vmware.2"
+input_entity_capiYaml_control_plane_node_pool_template        = "Ubuntu 20.04 and Kubernetes v1.26.8+vmware.2"

-input_entity_capiYaml_worker_node_pool_version         = "v1.25.7+vmware.2"
+input_entity_capiYaml_worker_node_pool_version         = "v1.26.8+vmware.2"
-input_entity_capiYaml_worker_node_pool_template        = "Ubuntu 20.04 and Kubernetes v1.25.7+vmware.2"
+input_entity_capiYaml_worker_node_pool_template        = "Ubuntu 20.04 and Kubernetes v1.26.8+vmware.2"

-input_entity_capiYaml_dns_imageTag           = "v1.9.3_vmware.8"
+input_entity_capiYaml_dns_imageTag           = "v1.9.3_vmware.11"
-input_entity_capiYaml_etcd_imageTag          = "v3.5.6_vmware.9"
+input_entity_capiYaml_etcd_imageTag          = "v3.5.6_vmware.14"

adambarreiro commented 9 months ago

Yes, it looks good to me. Now the input_entity just needs to be updated:

{
  "apiVersion": "capvcd.vmware.com/v1.1",
  "kind": "CAPVCDCluster",
  "name": "${name}",
  "metadata": {
     ...omitted
  },
  "spec": {
    "vcdKe": {
      ...omitted
    },
    "capiYaml": ${capi_yaml} <--- Updated YAML
  },
  "status": { ... } <-- Important to not miss this one: The object from the 'computed_entity' attribute that we need
}

vmware / terraform-provider-vcd