sbueringer
commented
1 year ago Thx for opening the issue. This sounds like a good idea to me in general. It should speed up a lot of provider tests and make them simpler regarding the preload.
I only see one edge case. Today it's possible to use a pre-existing Kubernetes cluster instead of a kind cluster (in the core Cluster API tests this is exposed via --e2e.use-existing-cluster). This means that today we don't have a hard dependency to kind, by hard-coding usage of kind in the clusterctl upgrade test we would make it a hard dependency.
So I would suggest to add an additional field to ClusterctlUpgradeSpecInput to make it configurable if the secondary mgmt cluster should be created with the infra provider or with kind.
fabriziopandini
k8s-ci-robot
k8s-triage-robot
lentzi90
User Story
As an infrastructure provider developer I would like to use a less resource intensive and time consuming secondary management cluster in clusterctl upgrade tests to avoid resource congestion and long run times.
Detailed Description
Currently the clusterctl upgrade test is making use of a secondary management cluster. My understanding is that this is done to be able to treat the test similar to all other tests even though it is using older controller versions to start with. If we used the bootstrap cluster directly it would affect all other tests that potentially run in parallel with it.
Unfortunately this secondary management cluster can be an issue for providers. The bootstrap cluster is a light weight kind cluster that makes it easy to load any images that may be needed for the test. But the secondary management cluster will be a "normal provider cluster", meaning that it 1) may require way more resources (VMs vs containers) and 2) cannot easily load the images.
Taking CAPO as example, the secondary management cluster would consist of 3 VMs (1 bastion, 1 control plane and 1 worker). Add to this the actual workload cluster with 1 more bastion, 1 control plane and 2 workers when scaled. A total of 7 VMs for this one test and then we are not counting other cloud resources like load balancers. We also need a way to get the controller image to the secondary management cluster since we cannot use
kind loadin this case.I would like to reuse something like CreateKindBootstrapClusterAndLoadImages for the secondary management cluster instead. Basically replacing this with this.
Anything else you would like to add:
It is very possible that I'm missing some reason why the secondary management cluster is created as a workload cluster from the provider. Please enlighten me in that case! :slightly_smiling_face:
/kind feature