gotestyourself / gotestsum

'go test' runner with output optimized for humans, JUnit XML for CI integration, and a summary of the test results.
Apache License 2.0
1.99k stars 118 forks source link

Duplicated `go test` arguments on rerun, causing `package is not in std` #381

Closed matthiasr closed 8 months ago

matthiasr commented 8 months ago

Given a failing test

test_test.go ```go package buildtag import ( "testing" ) func TestTest(t *testing.T) { t.Error("failing") } ```

when running gotestsum with multiple reruns and additional go test flags:

gotestsum --rerun-fails=4 --packages=./... -- -run TestTest -tags some_tag -json

we get the red-herring error

package some_tag is not in std (/opt/homebrew/Cellar/go/1.21.3/libexec/src/some_tag)

on the second(!) rerun.

full output ```log ✖ . (109ms) DONE 1 tests, 1 failure in 0.245s ✖ . (65ms) DONE 2 runs, 2 tests, 2 failures in 0.444s package some_tag is not in std (/opt/homebrew/Cellar/go/1.21.3/libexec/src/some_tag) === Failed === FAIL: . TestTest (0.00s) tag_test.go:8: failing === FAIL: . TestTest (re-run 1) (0.00s) tag_test.go:8: failing === Errors package some_tag is not in std (/opt/homebrew/Cellar/go/1.21.3/libexec/src/some_tag) DONE 3 runs, 2 tests, 2 failures, 1 error in 0.449s ERROR rerun aborted because previous run had errors ```

Note that the reruns stop after 2, even though I specified 4.

I set up an interception of the go commands that gotestsum actually invokes, and these are interesting:

go test -run TestTest -tags some_tag -json ./...
go test -test.run=^TestTest$ -tags some_tag -json buildtag
go test -test.run=^TestTest$ -tags some_tag -json some_tag -json buildtag
how to intercept Add a file `go` to some directory, say `./fake-go`, with the contents ```sh #!/bin/sh echo go "$@" >> commands.log exec /opt/homebrew/bin/go "$@" ``` If not using homebrew, insert the absolute path to `go` (`command -v go`). Make it executable (`chmod +x fake-go/go`) and run gotestsum as ```sh PATH="$(pwd)/fake-go:${PATH}" gotestsum … ```

Something goes wrong when massaging the test arguments for reruns, but only the second time. I can work around this by using a one-argument form of the tags flag:

gotestsum --rerun-fails=4 --packages=./... -- -run TestTest -tags=some_tag -json

which correctly reruns.

full output ```log ✖ . (231ms) DONE 1 tests, 1 failure in 0.710s ✖ . (91ms) DONE 2 runs, 2 tests, 2 failures in 0.947s ✖ . (84ms) DONE 3 runs, 3 tests, 3 failures in 1.167s ✖ . (74ms) DONE 4 runs, 4 tests, 4 failures in 1.372s ✖ . (87ms) === Failed === FAIL: . TestTest (0.00s) tag_test.go:8: failing === FAIL: . TestTest (re-run 1) (0.00s) tag_test.go:8: failing === FAIL: . TestTest (re-run 2) (0.00s) tag_test.go:8: failing === FAIL: . TestTest (re-run 3) (0.00s) tag_test.go:8: failing === FAIL: . TestTest (re-run 4) (0.00s) tag_test.go:8: failing DONE 5 runs, 5 tests, 5 failures in 1.593s 0.84s user 1.34s system 131% cpu 1.646s total ```

The commands it invokes are still wrong but this no longer breaks go test:

go test -run TestTest -tags=some_tag -json ./...
go test -test.run=^TestTest$ -tags=some_tag -json buildtag
go test -test.run=^TestTest$ -tags=some_tag -json -tags=some_tag -json buildtag
go test -test.run=^TestTest$ -tags=some_tag -json -tags=some_tag -json buildtag
go test -test.run=^TestTest$ -tags=some_tag -json -tags=some_tag -json buildtag

It is worth noting that the argument list does not continue to grow after the second rerun.

dnephin commented 8 months ago

Thank you for the bug report! You can tell gotestsum to print the commands it is running with --debug as well. That may be easier than faking the go command.

It seems like there is probably an append bug here:

https://github.com/gotestyourself/gotestsum/blob/b8d5d8fc4f23b3be97c5244b1ccded5e2b18fc47/cmd/main.go#L358

That modifies args, which also modifies the backing array of opts.args and opts.args is re-used again later.

This line: https://github.com/gotestyourself/gotestsum/blob/b8d5d8fc4f23b3be97c5244b1ccded5e2b18fc47/cmd/main.go#L338

Should make a copy instead:

args := append([]string{}, opts.args...)