Testing controllers with envtest cluster

[erikgb]

Working with trivy-operator for a couple of weeks, I already managed to propose changes that introduced bugs. And these bugs were not detected by CI and merged into main. That tells me we need better tests. 😄

I am unable to run integration tests locally for two reasons:

1. My development environment is docker-based, and kind does not run on docker (dind). I've been using k3d as a workaround, which can run on docker as an alternative, but there is always a risk when having to modify local files to run things.
2. Because I work in an air-gapped environment, trivy (scanner) is unable to update the DB without modifications to trivy-operator configuration.

To mend this, I am suggesting to introduce envtest clusters to run (integration) tests. How this might look like, is fairly well documented in the kubebuilder book. This will allow the integration tests to run in a standard Go test - without bootstrapping a full cluster (like kind).

• Running tests using make will be supported without requirements to install anything, as there is a nice tool to setup envtest available. Running tests for multiple versions of Kubernetes is also possible with ease.
• For running tests from a local development environment (IDE) you would have to install kubebuilder-tools, which is a small package containing kubectl, etcd and api-server.

I can prepare a PR suggesting envtest integration tests for vulnerability scanning, so you can have a look at the details in this proposal. We (as in my team/company) are currently only interested in the vulnerability scanning feature in trivy-operator, so I am not prepared to do the work on the other controllers. But if the community likes this idea, I hope that someone else can contribute writing tests for the remaining controllers!

If you think this approach looks good, I think this could be an overall plan for refactoring/improving the tests in trivy-operator:

1. Write envtest cluster tests for all controllers in trivy-operator. Since we already have unit tests (that will not change), I think these tests should be fairly simple, at least in the beginning.
2. Introduce e2e (end-to-end) tests to run tests against a real cluster (kind). The goal for the initial version of these tests will be to be good enough to delete the existing integration tests (next task). We (my team) are using kuttl to drive internal operator e2e-tests, but I have no strong feelings about tooling here. It works well for us, and allows us to write good tests with little effort. We can decide on the tooling later.
3. Delete the existing integration tests, which I think are confusing and hard to use/improve.

[chen-keinan]

Great initiative looking forward to see the PR

[erikgb]

Great initiative looking forward to see the PR

@chen-keinan I am glad you like the idea! Here's a teaser (spoiler-alert): https://github.com/aquasecurity/trivy-operator/pull/337/files ;-)