defenseunicorns / maru-runner

The Unicorn Task Runner
Apache License 2.0
11 stars 1 forks source link

fix: sort slices before comparing in test #142

Closed catsby closed 2 months ago

catsby commented 2 months ago

Description

The Runner test TestRunner_GetBaseActionCfg/extraEnv_adds_and_overrides_defaults in src/pkg/runner/actions_test.go is flaky because we don't sort the string slices before comparing them, and occasionally they get created in slightly different orders:

=== RUN   TestRunner_GetBaseActionCfg/extraEnv_adds_and_overrides_defaults
    actions_test.go:502:
                Error Trace:    /Users/clint/go/github.com/defenseunicorns/maru-runner/src/pkg/runner/actions_test.go:502
                Error:          Not equal:
                                expected: []string{"ENV1=fromDefault", "ENV2=xyz1", "ENV4=fromSet", "ENV2=alsoFromEnv", "ENV3=fromExtra"}
                                actual  : []string{"ENV1=fromDefault", "ENV2=xyz1", "ENV4=fromSet", "ENV3=fromExtra", "ENV2=alsoFromEnv"}

note the expected "ENV2=alsoFromEnv", "ENV3=fromExtra" vs. the actual "ENV3=fromExtra", "ENV2=alsoFromEnv".

Assuming a successful test run produces this output:

=== RUN   TestRunner_GetBaseActionCfg
=== RUN   TestRunner_GetBaseActionCfg/extraEnv_adds_and_overrides_defaults
--- PASS: TestRunner_GetBaseActionCfg (0.00s)
    --- PASS: TestRunner_GetBaseActionCfg/extraEnv_adds_and_overrides_defaults (0.00s)

If we run the test enough times it will fail, as shown below where we would expect 200 lines of -- PASS here when running the test 100 times (2x per run), and 0 for --- FAIL:

 github.com/defenseunicorns/maru-runner on  main [$]
 ➜ go test ./src/pkg/runner/... -v -run=TestRunner_GetBaseActionCfg/extraEnv_adds_and_overrides_defaults -count=100 | grep '\-\-\- PASS' | wc -l
     142

 github.com/defenseunicorns/maru-runner on  main
 ➜ go test ./src/pkg/runner/... -v -run=TestRunner_GetBaseActionCfg/extraEnv_adds_and_overrides_defaults -count=100 | grep '\-\-\- FAIL' | wc -l
      46

It's flaky, so your numbers will vary each time you run it.

In this PR we sort both slices before comparing. Here's the updated output:

 github.com/defenseunicorns/maru-runner on  fix-runner-test-sort
 ➜ go test ./src/pkg/runner/... -v -run=TestRunner_GetBaseActionCfg/extraEnv_adds_and_overrides_defaults -count=100 | grep '\-\-\- PASS' | wc -l
     200

 github.com/defenseunicorns/maru-runner on  fix-runner-test-sort
 ➜ go test ./src/pkg/runner/... -v -run=TestRunner_GetBaseActionCfg/extraEnv_adds_and_overrides_defaults -count=100 | grep '\-\-\- FAIL' | wc -l
       0

Type of change

Checklist before merging

ericwyles commented 2 months ago

Thanks for this!