Automated testing of default apps

yulianedyalkova commented 6 months ago

Currently pausing this because of this. The main issues are:

[ ] We're planning to remove default apps repo as part of https://intranet.giantswarm.io/docs/product/releases/capi-release-framework/ and it's not quite clear how flow will change.
[ ] App names are inconsistent across collections (can potentially get fixed through configuration).

AverageMarcus commented 3 months ago

We discussed this partially last week at our on-site. Here's some points from those discussions:

Team Turtles is currently in the progress of refactoring how / where default apps are defined and moving them from the various default-apps-XXX Apps into the cluster-XXX and cluster charts.
As part of this refactoring there should be a property that allows overriding the app version / catalog that we can use for the tests. There is currently an outstanding work to also have this support the HelmReleases though (see here)
The exact map key needed to override the values is still up for discussion but will hopefully match the App name.
It's still not clear yet how to work out if an app is a default app or not. As some apps may be default for only a specific provider we could do with having some way to automatically work it out but not sure if that will be possible so it might need to be configured in the config.yaml of the app tests.
We might need to rebuild the Pipeline to be able to handle overriding the App version / catalog before the cluster creation as that is handled outside of the test framework. This would need to also then know which test suites are upgrades and which aren't - something that is currently only known within the test suite code itself. ℹ This point needs more thought and planning.

AverageMarcus commented 3 months ago

Once cluster creation and deletion is moved into apptest-framework we will have more control over the default apps created with the cluster.

[ ] Create a function that templates a given cluster App and checks if any of the included Apps (or HelmReleases) match the App we are trying to test.
[ ] Update the BeforeSuite logic to perform the above check for if it is a default app and if so modify the Cluster values to override the version and catalog
[ ] Implement an update function that modifies the version in the chart values and then watches for the upgrade to complete.

⚠️ Blockers:

AverageMarcus commented 2 weeks ago

Some thoughts now that default apps are defined via Releases and how this impacts testing:

We're able to check if a given app is a default app or not by checking if it exists in Release.Spec.Apps[]
Default app versions can be overridden by modifying the values in Release.Spec.Apps[] but this is limited in what can be set (e.g. no install namespace or chart values)
Upgrades must be performed by upgrading the Release
For this to be possible in code we'd need to update cluster-standup-teardown to have a Standup function that can take in a BuiltCluster that also defines the Release
Apps within bundles that are default apps are a pain as I don't think there is a way on the Release to override the version of the child apps. (Do bundles even make sense for default apps on a Release???)
The test suite will need to lookup the App from the MC after cluster creation so the tests have access to the correct App in the test state.

giantswarm / roadmap