Fetch charts under test prior to running tests

[viccuad]

helm has helm fetch, that obtains a chart from a URL or repo, and saves it as tar.gz. For all operations like helm install, helm upgrade, helm accepts a local folder or a tar.gz as a chart.

I would like to separate the setup of the end-to-end tests and the run of tests, where:

• First, we helm fetch all needed charts, from whatever we have defined. For example, we can support as input a yaml file, or env vars, etc, where we list charts upstream names, their location (a repo, a url for a tar.gz, a folder, etc), and our test name for them: (kubewarden-controller-1.7.0, https://charts.kubewarden.io, kubewarden-controller): fetch the kubewarden-controller-1.7.0 chart from that url and save it as kubewarden-controller.
• A simple "test setup" script can naively iterate through all of those and perform a helm fetch --untar, or helm fetch --untar --devel if it has -rc on the name. This script doesn't need much checks, just iterate through everything and do the helm fetch.
• The result of the script is that we stored all those helm charts in a folder. So we would have something similar to:

  charts-under-test
  ├── cert-manager
  ├── kubewarden-controller-ga
  ├── kubewarden-controller
  ├── kubewarden-controller-next
  ├── kubewarden-defaults
  └── kubewarden-defaults-next

• Now, we consume those charts, with those folder names, for our tests. That means, if we are testing the kubewarden-controller-1.7.0-rc1, it got downloaded and extracted at kubewarden-controller. We can use that chart name in all our tests.
  • If we have a test that installs kubewarden-controller-1.7.0-rc1 and runs some tests, it performs helm install kubewarden-controller && run test.
  • If we have a test that checks that the kubewarden-controller-1.7.0-rc1 is upgradable to a yet to be released version, we would perform then helm upgrade kubewarden-controller kubewarden-controller-next.
  • If we have a test that checks that we can upgrade from the latest GA of kubewarden-controller to kubewarden-controller-1.7.0-rc1, we perform helm install kubewarden-controller-ga && helm upgrade kubewarden-controller.

I think this brings some nice behaviour:

• One can know exactly which versions of charts are under test, by looking at the yaml config file.
• By having this new fetch phase separated, we can fetch specific charts in CI easily. We could use also helm plugins such as helm-git to fetch the needed helm-charts from a PR branch, to test them.
• CI can use update-cli to open PRs that bump listed versions of charts in the yaml config file. For example, bumping to latest rancher chart, cert-manager, or test utilities.
• One can perform this fetch phase first, by hand, possibly seeding the charts-under-test folder manually. This is of great help for local development: I can easily consume a helm chart from a git branch, and consume my local helm chart folder by making a symlink into the charts-under-test folder.
• By fetching all charts under test before, we will fail early if some of them are misconfigured or not reachable, instead of mid tests which is more expensive.

Improvements

A somewhat similar approach could be used for configuring the charts. For example, a specific combination of chart versions may need to consume a container image :latest and set a specific config. It would be easier to drop a default-values-chart-under-test/values.yaml with:

auditScanner
  image
    tag: latest
  serviceAccountName: my-development-sa

This can help when using the end-to-end tests when developing, as one can just drop the values.yaml, and the helm-chart. The same approach can be used recursively inside of the tests, with each test being a folder, where there's resources to kubectl apply, values.yaml to upgrade to, etc.

[viccuad]

As talked on daily, piling up here several features wanted.

The majority of the tests are tightly coupled with the helm chart code, hence the test code could be part of the chart repo. The only tests not tightly coupled would be for chart dependencies (such as open-telemetry, grafana, etc), and we only need minimum integration tests for those. For vendored chart dependencies such as policy-reporter-ui are tightly coupled and we can include integration tests in the chart repo.

My list of desired features:

• Allow testing any chart, from targz, local, or tag (the original issue description) Could be implemented by migrating to helm/chart-testing. Looks mature enough, but maintainers a bit AWOL (see here and here).
• Have chart unit tests that check that X values.yml + Y template gives us the desired Z rendered YAML. Could be implemented by:
  • https://github.com/helm-unittest/helm-unittest
  • Expanding on the values schema that we have, plus helm lint or chart-testing lint.
  • Lint the resulting Kubernetes rendered manifests with a tool such as https://github.com/yannh/kubeconform, kubesec, kubescape, etc.
  • Note, templates are not rendered unless there's a values.yaml that exercises them. Hence, for each feature, provide examples of values.yaml, shipped inside of the chart, that will exercise the correct templating.
• Integration tests. Once the chart is installed, check that it's working as expected. Could be implemented by creating <chart>/templates/tests/ entries. See https://helm.sh/docs/topics/chart_tests/. This allows for jobs to run on a cluster with the chart installed. This ships with the chart, can be run by users to certify the chart is working as intended.
• Dependencies integration tests, for cert-manager, opentelemetry, policy-reporter-ui, etc.
• Airgap tests
• If we have done everything correctly, all these features work under airgap

Example of a Helm chart with unit, integration tests: https://github.com/hashicorp/vault-helm, albeit DIY.