run acceptance tests with multiple versions of kubernetes

[coryflucas]

Description

Leverage a matrix to run the acceptance tests against multiple versions of Kubernetes.

The versions are based on the default version previously used (1.27) and the versions used in the manifest acceptance tests but with corrected SHAs to match the version of kind in use. Note: there is no 1.30 image available for kind 0.20 (its available for 0.23) so I opted to use 1.29.

Acceptance tests

N/A

Release Note

Release note for CHANGELOG:

NONE

References

fixes #2562

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[BBBmau]

can you provide a link to your workflow of your changes to confirm? We should be able to see the matrix of the k8s versions all passing.

[coryflucas]

I had originally thought the workflow had already run in my branch but it hadn't due to the workflow being restricted to only run against the Hashicorp org. I got it running and have found that a subset of the tests fail against older versions of Kubernetes. The failures seem to fall into 3 categories:

1. They are covering features that aren't supported in those k8s versions.
2. They are testing things that are both supported and some that are not supported in those k8s versions, so really just the tests need refactoring.
3. The provider has unexpected behavior with older versions k8s. The number of issues in this category looks to be pretty small (it looks like 3 right now) and they are relatively minor (mostly they result in an apply followed by a plan still showing changes).

I'd like to propose the following to get the tests in a passing state:

• For all 3 categories of tests add a skipIfClusterVersionLessThan precheck to the tests. This resolves category 1, and gets the category 2 and 3 tests back to a passing state and still has them running against more versions of k8s than they are running against today.
• For tests in category 2 open an issue with the list of tests and the versions they should be able to work under which I'm happy to pick up as a follow up to this. I'm thinking a single issue for this should be ok given it should just be test refactoring. I had started trying to do this in my current branch but it got to a bit larger than I expected pretty quickly.
• For tests in category 3 open an issue each individual problem. Again I am happy to start looking at these as well if we deem them worth fixing.

Let me know what you think about this approach and I'll follow up.

[coryflucas]

@BBBmau I wen't ahead and did my proposed change in a new branch and have all tests now passing. Here is a break down of the issues I found:

• 4 tests were missing version checks when testing features that are not available on older clusters.
• 4 tests need refactoring to split testing features that are not available on older clusters out to new tests.
• 2 tests have failures due to provider issues. Both are issues where after an apply a plan still shows changes to be applied on older clusters versions.
• 2 tests that only run for old versions of k8s had actual bugs that I fixed as well.

Here is a link to the workflow run with the changes + a change to enable the workflow in my fork. Please let me know if you'd like me to merge those changes into this branch or go with a different approach.

[BBBmau]

@coryflucas thanks for working on this! I haven't had the change to really look at your PR in depth. I plan on doing this next week just as a heads-up. Apologies for the delay!