Improve e2e testing

[sbueringer]

Improve e2e tests

We compared CAPV govmomi & supervisor e2e tests with core CAPI e2e tests. As a result we want to make the following improvements.

Adding e2e tests:

• [x] P0: Add v1.30 => v1.31 upgrade e2e test + check ci-latest (supervisor+govmomi) (@sbueringer)
  • PR: https://github.com/kubernetes/test-infra/pull/32610
  • PR: https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/pull/2999
  • Plan is to add more and more upgrades jobs for new Kubernetes versions until we have the same test coverage as core CAPI.
• [x] P0: Add test "When testing ClusterClass changes [ClusterClass]" (supervisor) (@zhanggbj)
  • PR: https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/pull/3011
• [ ] P1: Add test "Ensure OwnerReferences and Finalizers are resilient" (supervisor) (@fabriziopandini)
  • [x] CAPI PR https://github.com/kubernetes-sigs/cluster-api/pull/10693
  • [ ] CAPV PR https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/pull/3059
• [x] P1: Also use ValidateResourceVersionStable func (https://github.com/kubernetes-sigs/cluster-api/pull/10530) to ownerRef & finalizer tests (supervisor+govmomi) (@fabriziopandini) https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/pull/3033
• [ ] P1: Let's check if we have unit test coverage for the following tests for supervisor @chrischdi
  • capv-e2e.[It] Hardware version upgrade creates a cluster with VM hardware versions upgraded
  • capv-e2e.[It] Label nodes with ESXi host info creates a workload cluster whose nodes have the ESXi host info
• [ ] P1: Add test "When testing ClusterClass rollouts [ClusterClass]" (supervisor) (@zhanggbj)
  • PR: https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/pull/3023
• [ ] P1: Add clusterctl upgrade tests n-2, n-1 (supervisor) (@zhanggbj)
  • CAPV 1.9=>current, CAPI 1.6=>1.7, CAPV 1.10=>current, CAPI 1.7=>1.7 (same we already have on main for govmomi)
  • PR: https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/pull/3024
• [ ] P1: When upgrading a workload cluster using ClusterClass with RuntimeSDK [ClusterClass] (supervisor) @chrischdi
  • This requires implementing a minimal Runtime Extension (let's discuss before we start implementing it, and let's do the other more straightforward sub-tasks first)
  • CAPI PR: https://github.com/kubernetes-sigs/cluster-api/pull/10788
  • CAPV PR: https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/pull/3069
• [ ] P2: Add test "When testing the machinery for scale testing" (vcsim + supervisor)
• [ ] P2: Add clusterctl upgrade tests n-3 (supervisor)
  • Same as above just go one version more back
• [ ] P?: Audit if we can run more tests with vcsim

Other improvements:

• [x] Use a kind cluster for clusterctl upgrade tests CAPI#7613 (@fabriziopandini)
  • [x] CAPI PR https://github.com/kubernetes-sigs/cluster-api/pull/10639
  • [x] CAPV PR https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/pull/3050
• [ ] Introduce Ginkgo labels (CAPI+CAPV): CAPI#9620
  • Let's discuss details on how to do this first

Reconsider if we need it later:

• Add test: When using the autoscaler with Cluster API using ClusterClass (incl. scale from zero coverage?)

govmomi has the following additional jobs (intentionally):

* accepted: it's okay if it's only tested with govmomi: * capv-e2e.[It] Cluster creation with [Ignition] bootstrap [PR-Blocking] Should create a workload cluster * accepted: not surfaced in CAPV supervisor CRDs: * capv-e2e.[It] Cluster creation with anti affined nodes should create a cluster with anti-affined nodes * capv-e2e.[It] Cluster creation with storage policy should create a cluster successfully * capv-e2e.[It] DHCPOverrides configuration test when Creating a cluster with DHCPOverrides configured Only configures the network with the provided nameservers * accepted: IPAM provider not supported in supervisor mode * capv-e2e.[It] ClusterClass Creation using Cluster API quick-start test and IPAM Provider [ClusterClass] Should create a workload cluster

[fabriziopandini]

/area supervisor Given that most of the improvements are for supervisor mode

[zhanggbj]

I would like to start with

• [x] P0: Add test "When testing ClusterClass changes [ClusterClass]" (supervisor) (#3011 )

[zhanggbj]

Also would like to pick up a similar one :-)

• [ ] P1: Add test "When testing ClusterClass rollouts [ClusterClass]" (supervisor)

[zhanggbj]

It seems this one is more about testing efforts. Just raised a PR to test.

• [ ] P1: Add clusterctl upgrade tests n-2, n-1 (supervisor)

[sbueringer]

Yeah good case is that most if it just works once added :)

A few require a bit more work

[chrischdi]

Taking this one

• [ ] P1: Let's check if we have unit test coverage for the following tests for supervisor