The Runner test TestRunner_GetBaseActionCfg/extraEnv_adds_and_overrides_defaults in src/pkg/runner/actions_test.go is flaky because we don't sort the string slices before comparing them, and occasionally they get created in slightly different orders:
=== RUN TestRunner_GetBaseActionCfg/extraEnv_adds_and_overrides_defaults
actions_test.go:502:
Error Trace: /Users/clint/go/github.com/defenseunicorns/maru-runner/src/pkg/runner/actions_test.go:502
Error: Not equal:
expected: []string{"ENV1=fromDefault", "ENV2=xyz1", "ENV4=fromSet", "ENV2=alsoFromEnv", "ENV3=fromExtra"}
actual : []string{"ENV1=fromDefault", "ENV2=xyz1", "ENV4=fromSet", "ENV3=fromExtra", "ENV2=alsoFromEnv"}
note the expected "ENV2=alsoFromEnv", "ENV3=fromExtra" vs. the actual "ENV3=fromExtra", "ENV2=alsoFromEnv".
Assuming a successful test run produces this output:
=== RUN TestRunner_GetBaseActionCfg
=== RUN TestRunner_GetBaseActionCfg/extraEnv_adds_and_overrides_defaults
--- PASS: TestRunner_GetBaseActionCfg (0.00s)
--- PASS: TestRunner_GetBaseActionCfg/extraEnv_adds_and_overrides_defaults (0.00s)
If we run the test enough times it will fail, as shown below where we would expect 200 lines of -- PASS here when running the test 100 times (2x per run), and 0 for --- FAIL:
github.com/defenseunicorns/maru-runner on main [$]
➜ go test ./src/pkg/runner/... -v -run=TestRunner_GetBaseActionCfg/extraEnv_adds_and_overrides_defaults -count=100 | grep '\-\-\- PASS' | wc -l
142
github.com/defenseunicorns/maru-runner on main
➜ go test ./src/pkg/runner/... -v -run=TestRunner_GetBaseActionCfg/extraEnv_adds_and_overrides_defaults -count=100 | grep '\-\-\- FAIL' | wc -l
46
It's flaky, so your numbers will vary each time you run it.
In this PR we sort both slices before comparing.
Here's the updated output:
github.com/defenseunicorns/maru-runner on fix-runner-test-sort
➜ go test ./src/pkg/runner/... -v -run=TestRunner_GetBaseActionCfg/extraEnv_adds_and_overrides_defaults -count=100 | grep '\-\-\- PASS' | wc -l
200
github.com/defenseunicorns/maru-runner on fix-runner-test-sort
➜ go test ./src/pkg/runner/... -v -run=TestRunner_GetBaseActionCfg/extraEnv_adds_and_overrides_defaults -count=100 | grep '\-\-\- FAIL' | wc -l
0
Type of change
[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
Description
The Runner test
TestRunner_GetBaseActionCfg/extraEnv_adds_and_overrides_defaults
insrc/pkg/runner/actions_test.go
is flaky because we don't sort the string slices before comparing them, and occasionally they get created in slightly different orders:note the expected
"ENV2=alsoFromEnv", "ENV3=fromExtra"
vs. the actual"ENV3=fromExtra", "ENV2=alsoFromEnv"
.Assuming a successful test run produces this output:
If we run the test enough times it will fail, as shown below where we would expect 200 lines of
-- PASS
here when running the test 100 times (2x per run), and 0 for--- FAIL
:It's flaky, so your numbers will vary each time you run it.
In this PR we sort both slices before comparing. Here's the updated output:
Type of change
Checklist before merging