Improve e2e testing - Githubissues

loft-sh / vcluster

vCluster - Create fully functional virtual Kubernetes clusters - Each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.

https://www.vcluster.com

Apache License 2.0

6.32k stars 403 forks source link

Improve e2e testing #503

Open FabianKramm opened 2 years ago

FabianKramm commented 2 years ago

Currently we deploy vcluster in a single configuration in parallel in 4 different jobs for each distro. While this works and smoke tests vcluster, several features and areas of vcluster remain untested due to this single deployment of vcluster. Hence I propose to deploy vcluster within those jobs multiple times in different configurations, so that we can test certain features as well.

Currently untested vcluster functionality (might be incomplete):

[ ] default installation with namespace admin RBAC only
[x] rootless mode
[x] enable scheduler
[x] sync all nodes
[ ] vcluster in vcluster
[x] sync nodes based on node selector
[x] isolated mode
[x] target namespace
[x] pause / resume vcluster
[ ] persistentvolumes
[ ] storageclasses
[ ] priorityclasses
[ ] serviceaccounts
[ ] plugins

matskiv commented 2 years ago

Me and @pratikjagrut discussed a possible refactor of GH actions. Proposed new setup:

a build action which will build the syncer image and push it to a public accessible image repo with PR number as the tag - PR1234. This has an added benefit of easier testing of changes before merging, e.g. issue reporter can use this image to validate the fix before it is merged.
a matrix of e2e tests actions will start a KinD cluster, build vcluster CLI from source code and wait until the image is available in the image repo. Then freshly built vcluster CLI will be used to install vcluster. Then the tests will be executed.

I don't know if we should use the same repo as we have for release builds. - @FabianKramm what do you think? If not, then I would personally use quay.io for hosting images. It doesn't have unauthorized pull restrictions which are present in docker hub. Also, it is possible to set expiration time - automatic deletion of an image after a certain time.

FabianKramm commented 2 years ago

Thanks for the update @matskiv ! Regarding your ideas:

Sounds great! One question regarding cleanup: would we have an automated cleanup job as soon as the PR is merged as well? Or would we tidy up the PR images from time to time ourselves?
Thats a very good idea, this way we would also test the vcluster cli as well.

We can have a separate package such as ghcr.io/loft-sh/vcluster-dev for PR builds while ghcr.io/loft-sh/vcluster for release builds.

matskiv commented 2 years ago

@FabianKramm We would have an automated clean-up job that will trigger on PR merge. A separate ghcr.io/loft-sh/vcluster-dev package sounds good to me.

pratikjagrut commented 1 year ago

There are a few more tests remaining from the description. I think we should keep it open.