added modular tests for cluster with topology for all supported k8s versions

[deepakm-ntnx]

What this PR does / why we need it: Goal

• Make it easy to add custom e2e tests
• Move all test config/parameters from prow job to code
• Its unclear from test code as to which variables from nutanix.yaml file are used. We need to make it explicit in the code
• Make all the tests run with all supported k8s version explicitly instead of depending on prow config manipulation for CI
• Use Golang option pattern to add parameters without changing the base methods and make it extensible

TODO

• Allow easy functions to install addons/apps/services so that we can run all version with all the apps like kubebench, kubeburner, nutanix services etc for e2e testing
• add more negative test cases

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged): Fixes #

How Has This Been Tested?:

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration and test output

Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

Release note:

[nutanix-cn-prow-bot[bot]]

@deepakm-ntnx: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-capx-conformance	f4c0dd4089769ea9e25c647bf2b2b1dd2575e443	link	false	/test e2e-capx-conformance
ci/prow/e2e-k8s-upgrade	f4c0dd4089769ea9e25c647bf2b2b1dd2575e443	link	false	/test e2e-k8s-upgrade
ci/prow/e2e-capx-controller-upgrade	f4c0dd4089769ea9e25c647bf2b2b1dd2575e443	link	false	/test e2e-capx-controller-upgrade

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).

[deepakm-ntnx]

observed following test failure

• [FAILED] [447.442 seconds]
When scaling down cluster with topology  [It] Scale down a cluster with CP and Worker node VCPUSockets from 3 to 2 with Kube127 [clusterclass, cluster-topology-scale-up-down]
/Users/deepak.muley/go/src/github.com/deepakm-ntnx/cluster-api-provider-nutanix/test/e2e/cluster_topology_scale_up_down_test.go:169

  [FAILED] Timed out after 300.001s.
  Expected success, but got an error:
      <*errors.errorString | 0xc00074e4a0>: {
          s: "Assertion in callback at /Users/deepak.muley/go/src/github.com/deepakm-ntnx/cluster-api-provider-nutanix/test/e2e/test_helpers.go:610 failed:\nexpected num sockets of VMs to be equal to 2 but was 3\nExpected\n    : 3\nto equal\n    : 2",
      }
      Assertion in callback at /Users/deepak.muley/go/src/github.com/deepakm-ntnx/cluster-api-provider-nutanix/test/e2e/test_helpers.go:610 failed:
      expected num sockets of VMs to be equal to 2 but was 3
      Expected
          : 3
      to equal
          : 2
  In [It] at: /Users/deepak.muley/go/src/github.com/deepakm-ntnx/cluster-api-provider-nutanix/test/e2e/test_helpers.go:618 @ 02/06/24 10:55:00.129

  Full Stack Trace

also all the 1.29 tests are failing

[deepakm-ntnx]

I think it looked good to me, but I'm still very green to all this. It'd be good for someone else to take a look at this

Some linting tests look like their failing, but I don't quite see why, and blackduck failing doesn't look related to this PR

Thanks. ran explicit gofumpt on the files to fix the issues

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 20.37%. Comparing base (d755db6) to head (a44d7b9). Report is 35 commits behind head on main.

:exclamation: Current head a44d7b9 differs from pull request most recent head 2c01a75. Consider uploading reports for the commit 2c01a75 to get more accurate results

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #374 +/- ## ========================================== - Coverage 25.63% 20.37% -5.26% ========================================== Files 19 17 -2 Lines 1303 1212 -91 ========================================== - Hits 334 247 -87 + Misses 969 965 -4 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[thunderboltsid]

observed following test failure

• [FAILED] [447.442 seconds] When scaling down cluster with topology [It] Scale down a cluster with CP and Worker node VCPUSockets from 3 to 2 with Kube127 [clusterclass, cluster-topology-scale-up-down] /Users/deepak.muley/go/src/github.com/deepakm-ntnx/cluster-api-provider-nutanix/test/e2e/cluster_topology_scale_up_down_test.go:169

[FAILED] Timed out after 300.001s. Expected success, but got an error: <*errors.errorString | 0xc00074e4a0>: { s: "Assertion in callback at /Users/deepak.muley/go/src/github.com/deepakm-ntnx/cluster-api-provider-nutanix/test/e2e/test_helpers.go:610 failed:\nexpected num sockets of VMs to be equal to 2 but was 3\nExpected\n : 3\nto equal\n : 2", } Assertion in callback at /Users/deepak.muley/go/src/github.com/deepakm-ntnx/cluster-api-provider-nutanix/test/e2e/test_helpers.go:610 failed: expected num sockets of VMs to be equal to 2 but was 3 Expected : 3 to equal : 2 In [It] at: /Users/deepak.muley/go/src/github.com/deepakm-ntnx/cluster-api-provider-nutanix/test/e2e/test_helpers.go:618 @ 02/06/24 10:55:00.129

Full Stack Trace also all the 1.29 tests are failing

1.29 tests are failing because 1.29 was not yet supported on upstream image builder until last week. That's why we are only running the v1.28 as the presubmit tests.

[deepakm-ntnx]

As discussed with @thunderboltsid we will add only-for-validation label to all the clusterclass tests except basic so that CI does not get affected and later streamline the labels and jobs so that we can jobs parallely as well as granular labels

[deepakm-ntnx]

moved to https://github.com/nutanix-cloud-native/cluster-api-provider-nutanix/pull/422 to consolidate

[deepakm-ntnx]

https://github.com/nutanix-cloud-native/cluster-api-provider-nutanix/pull/422