vanvalenlab / kiosk-console

DeepCell Kiosk Distribution for Kubernetes on GKE and AWS

https://deepcell-kiosk.readthedocs.io

Other

35 stars 6 forks source link

Improve robustness of unit and integration tests. #330

Open willgraf opened 4 years ago

willgraf commented 4 years ago

Is your feature request related to a problem? Please describe. The project now has good testing pipelines thanks to #281, #320, and #326. However, these unit and integration tests can still be improved.

Describe the solution you'd like

Unit tests (moved from #272):

[ ] Test Makefiles in /conf/tasks.
[ ] Validate the output of helmfile build with kubectl.
[ ] Validate the /conf/addons with kubectl.
[ ] Test the shell scripts in /scripts.
[ ] Move the prometheus rules into their own file and test them.
[x] Test sphinx documentation builds.

Integration Tests (moved from #281):

[ ] Test the state of the deployed cluster (what charts are deployed, what HPAs are up, etc.)
[ ] Find the public IP address and make sure it's reachable.
[ ] Potentially test a tf-serving deployment by sending a pre-defined image through a model.

willgraf commented 4 years ago

Related to this: Sometimes the prometheus-operator can take a long time to come to a Running status, and will fail the integration tests. We can either reduce the default timeout time for helm deployments (10 minutes is a pretty long time), or we can try to use travis_wait in the test, or a combination of both.

UPDATE: This issue has pretty much gone away since migrating to GitHub Actions.

willgraf commented 3 years ago

Test the state of the deployed cluster (what charts are deployed, what HPAs are up, etc.)

Several PRs that will be in v1.4.0 (#395, #396, #397, #400, #405) migrate YAML configuration files into the helm chart/helmfile (e.g. HPAs, cert-manager configurations, configmaps). With these PRs, there are no actively used /conf/addons or /conf/patches files.

This should move some of the testing burdens from testing a cluster "state" to testing the charts themselves.

willgraf commented 3 years ago

402 introduces docs testing.

willgraf commented 3 years ago

Now that we are using GitHub Actions, it seems like there are a few tools available that may assist in testing helm charts (and maybe helmfiles).

chart-testing: A tool for linting/testing charts against a remote repository.
kind-action: Set up a kubernetes instance as an Action.
chart-testing-action: Example running chart-testing on a kind cluster in a GitHub Action.