Add upgrade tests to end-to-end tests

[viccuad]

When performing upgrades, the stateful parts of the stack may get out of sync. We are lucky that we don't have a DB nor we need to do DB schema migrations but we still have "stateful" parts and interfaces that need to be backwards compatible:

• policies.yml, sources.yml, verification,yml, for the policy-server
• Integration with other stacks (cert-manager, prometheus, open-telemetry)
• CRDs deserve special thought, as they cannot be easily downgraded, nor removed (they would remove all the resources from the cluster)

Nomenclature:

• Version Under Test (VUT): that version of kubewarden-controller/helm chart/policy-server that is being tested, be it main, a PR, etc. Unreleased yet.
• VUT+1: an unexistent version, consisting of the current codebase under test (main, PR), plus future changes.

Note: we don't have yet a published document on supported releases or release methodology, so I am assuming that we only support upgrades from the last publicly available release (no matter if major, minor, patch version). Skipping versions when upgrade is not supported. No minor/patch version releases are shipped unless it is for the latest version. Downgrade is not supported.

Acceptance criteria

Run all the following:

• [ ] Test Upgrade from latest release to VUT (upgrade to latest): Install latest publicly available release. Install some polices. Perform upgrade from the latest publicly available version to the VUT. Check that:
  • During upgrade, policies are still in effect.
  • After upgrade, no policies have been lost.
  • After upgrade, one can deploy new policies successfully, and the policies work.
• [ ] https://github.com/kubewarden/kubewarden-controller/issues/138

• [ ] Test Upgrade from VUT onto VUT+1 (new versions maintain upgradability): Install VUT. Install some policies. Generate a new synthetic VUT+1:

  • Bump the version of VUT.
  • Ideally, edit the manifests/binary contained in VUT so they are different, by substituting a simple innocuous word for example in the CRDs. This will trick the Kube API and Helm into deploying them again instead of skipping them.

  Upgrade to VUT+1. Check that:

  • During upgrade, policies are still in effect.
  • After upgrade, no policies have been lost.
  • After upgrade, one can deploy new policies successfully, and the policies work.

[jvanz]

Blocked until we fix https://github.com/kubewarden/helm-charts/issues/52

[jvanz]

Blocked until we fix kubewarden/helm-charts#52

I'll work on tests using controller configuration for now.

[jvanz]

Removed from the development board until we decided what to do first. As we now have the testing arch RFC, we may change the priorities.