Closed jessica-hofmeister closed 6 months ago
One other interesting thing is that doing a server show
on one of the instances in the cluster shows us image_id: none
, while the UI shows the 1.27.4 image
| id | 48a735e5-f3ea-4e4c-8e00-fc33525582f3 |
| image | N/A (booted from volume) |
| imageRef | None |
| image_id | None |
| instance_name | None
We need to save the cluster template ID here:
https://github.com/vexxhost/magnum-cluster-api/blob/main/magnum_cluster_api/driver.py#L231-L256
Oh, this might be far more complicated, I think during an upgrade the update_cluster_status
might be running somewhere else and override the save that happens here:
We actually need to setup some lock or actually in the cluster update sync code to pull the cluster-template from the magnum resource..
after scaling down the magnum conductor to 1, we retested and the results are exactly the same: each node upgrades to the new kubernetes version, but the cluster itself still references the old template id
Create 2 cluster templates. One with ubuntu-2204-kube-v1.27.3 image and one with ubuntu-2204-kube-v1.27.4 image. Create a cluster with the 1.27.3 template. After it comes up healthy, upgrade it to the 1.27.4 image. Cluster reaches upgrade_complete status, but still references old template.
Relevant images:
Create a cluster template for 1.27.3:
Create a cluster template for 1.27.4:
Create a cluster using the test-v1.27.3 template:
Wait for the cluster to come up healthy:
List the openstack templates:
See what template the cluster has currently:
Upgrade the cluster to the test-v1.27.4 template:
Wait for the cluster to finish upgrading:
The actual instances show that they are using the test-v1.27.4 image See what template the cluster has currently:
Notice that it still thinks the current template is the test-v1.27.3 even though the upgrade to the test-v1.27.4 completed.
And now...the test that shows it breaks: Attempt to scale up the cluster to 2 worker nodes:
The cluster resize actually fails. The details after trying to resize are: Note the difference between
coe_version
andkube_tag