Closed ErikSchierboom closed 1 year ago
Files with Coverage Reduction | New Missed Lines | % | ||
---|---|---|---|---|
testrunner/ast.go | 16 | 74.11% | ||
<!-- | Total: | 16 | --> |
Totals | |
---|---|
Change from base Build 5467785147: | -0.9% |
Covered Lines: | 404 |
Relevant Lines: | 622 |
For context, nearly all test runners have this setup. A few older ones (the OGs!) don't have the exact same setup, so this is an aim to standardise.
Addendum: I know that the Go code also has golden tests, but AFAICT it doesn't build and use the Docker image to verify it.
This doesn't yet work due to some permissions issue. If anyone has an idea how to solve that, I'd be interested to hear :)
Fixed the issue!
As nobody seems around to review this, I'm gonna go ahead and merge this.
@ErikSchierboom I was on vacation but I'm sure this is fine, thanks! Just as a heads up, when we update the Go version, there are sometimes minor changes in the test result output, mostly something like whitespace changes. I think the test still helps to increase confidence regarding the docker the file etc as you wrote, just limiting the expectations a bit that we won't need to touch them at all.
Just as a heads up, when we update the Go version, there are sometimes minor changes in the test result output, mostly something like whitespace changes.
That's perfectly fine, it happens to many test runners. One easy thing to do is:
bin/run-tests-in-docker.sh
find tests -name results.json -execdir cp results.json expected_results.json \;
This will create a results.json
and then copy it to the expected_results.json
, which allows you to quickly update the expected_results.json
files with the latest version.
This PR adds smoke tests for some common scenarios.
The goal of the smoke tests is to increase confidence that the Docker image that is produced works as intended. The repo contains other tests, which are great, but what this PR adds is that it runs an integration test using the following setup:
tests
directory mapped into the containerbin/run.sh
) on each of the test cases defined in thetests
directorytests
directoriesIf successful, we then know that:
bin/run.sh
entrypoint inside the Docker image is setup correctly