vexxhost / magnum-cluster-api

Cluster API driver for OpenStack Magnum
Apache License 2.0
44 stars 20 forks source link

Need a way to "upgrade" cluster when kube version doesn't change #381

Closed jessica-hofmeister closed 4 months ago

jessica-hofmeister commented 4 months ago

Create 2 cluster templates. One with ubuntu-2204-kube-v1.27.4 image and one with tenant-specific-v1.27.4 image. Note that both images are using v1.27.4.Create a cluster with the ubuntu-2204-kube-v1.27.4 template. After it comes up healthy, upgrade it to the tenant-specific-v1.27.4 image. Cluster reaches upgrade_complete status, and references the new template, but the nodes do not start a rolling update to 'take' the updated image.

Relevant images:

openstack image list
+--------------------------------------+-------------------------------------+--------+ | ID | Name | Status | +--------------------------------------+-------------------------------------+--------+ | 4baeb5c2-42cb-457e-81cf-885b100c19a2 | ubuntu-2204-kube-v1.27.4 | active | | fb450ad1-31e2-4e39-ae6c-a569e7af184b | tenant-specific-v1.27.4 | active | +--------------------------------------+-------------------------------------+--------+

Create a cluster template for ubuntu-2204-kube-v1.27.4 :

openstack coe cluster template create \
--image 4baeb5c2-42cb-457e-81cf-885b100c19a2 \
--coe kubernetes \ --flavor m1.medium \
--master-flavor m1.medium \ --external-network public \ --master-lb-enabled \ --floating-ip-disabled \ --network-driver calico \ --docker-storage-driver overlay2 \ --label kube_tag=v1.27.4 \ --label boot_volume_size=40 \ --label boot_volume_type=rbd1 \ --label master_lb_floating_ip_enabled=false \ --label audit_log_enabled=true --label os_distro=ubuntu \ test-v1.27.4

Create a cluster template for tenant-specific-v1.27.4 :

openstack coe cluster template create \ --image fb450ad1-31e2-4e39-ae6c-a569e7af184b \ --coe kubernetes \ --flavor m1.medium \ --master-flavor m1.medium \ --external-network public \ --master-lb-enabled \ --floating-ip-disabled \ --network-driver calico \ --docker-storage-driver overlay2 \ --label kube_tag=v1.27.4 \ --label boot_volume_size=40 \ --label boot_volume_type=rbd1 \ --label master_lb_floating_ip_enabled=false \ --label audit_log_enabled=true \ --label os_distro=ubuntu \ tenant-specific-test-v1.27.4

Create a cluster using the test-v1.27.4 template:

openstack coe cluster create \ --cluster-template test-v1.27.4 \ --master-count 3 \ --node-count 5 \ --fixed-network dev-k8s \ --keypair svc-account \ --floating-ip-disabled \ test-cluster

Wait for the cluster to come up healthy:

List the openstack templates:

openstack coe cluster template list
+--------------------------------------+----------------+------+ | uuid | name | tags | +--------------------------------------+----------------+------+ | bd45f426-ec98-4891-a49d-a4d699cc0669 | test-v1.27.4 | None | | ab9adeca-fc87-4166-9aa2-eb8d75678e02 | tenant-specific-test-v1.27.4 | None | +--------------------------------------+----------------+------+

See what template the cluster has currently:

openstack coe cluster show test-cluster -f value -c cluster_template_id
bd45f426-ec98-4891-a49d-a4d699cc0669

Upgrade the cluster to the tenant-specific-test-v1.27.4 template:

openstack coe cluster upgrade test-cluster tenant-specific-test-v1.27.4 Request to upgrade cluster test-cluster has been accepted.

See what template the cluster has now:

openstack coe cluster show test-cluster -f value -c cluster_template_id ab9adeca-fc87-4166-9aa2-eb8d75678e02

This is the new template id, which is great. At this point however, I would expect the nodes to begin their rolling upgrade, but nothing happens. The instances are left as they are, obviously therefore still referencing the original image, not the imaged used by the new template. image image

I'm hoping there is a way to trigger actually creating new nodes that reference the new image during an upgrade like this - when the user doesn't wish to change kube versions (or can't because they are running the latest), but deems there is still value in using an updated image.