Open james-johnston-thumbtack opened 4 years ago
/cc @mpvl @josharian
While replacing
os.Stdout
might arguably not the best approach to testing a command-line tool, somebody did write a test that way, and in such a scenario, I feel that the test runner should exhibit predictable behavior when presented with "interesting" tests like this one.
I definitely agree that you shouldn't be replacing os.Stdout
like that. A similar issue is how calling os.Exit(0)
can make an entire package pass its tests when it definitely shouldn't.
I don't know what we can do here, but I don't think adapting go test -json
to such tests is a good idea.
The testing
package itself could save os.Stdout
and os.Stderr
and restore them at the end of each test.
Any parallel test that attempts to reassign those would be reported as a race, but that seems like a good thing to report anyway.
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
It should; I don't see why it wouldn't as the code I highlighted below is in the master branch.
What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
Create a test that overwrites os.Stdout, and then run it using
go test -json
.https://play.golang.org/p/zeonvI5FdUJ
Then run the test using
go test -json
.What did you expect to see?
The test should succeed: both the exit code should be 0, and the JSON output should indicate a successful test.
What did you see instead?
The JSON output indicates the test failed, yet the exit code is still 0, indicating success:
Notice the failure that is reported in the JSON:
"Action":"fail"
. This is all very ambiguous: did the test pass or fail? The exit code says one thing, but the JSON output says another.Further investigation
Notice if we run the above play link, we get this output:
However, if we write a "normal" test that does not tamper with Stdout, we get additional output: a final "PASS": https://play.golang.org/p/3Z8hY3rAfhj
Output:
It would appear that test2json is interpreting the lack of a final "PASS" as being a failure when converting it to JSON. However, the main "go test" command does not do similarly, and thus exits with a "successful" exit code. Therefore, we end up with a test that is both "passing" and "failing" simultaneously.
The issue appears to be that the
testing.go
file is completely written under the assumption that the end-user will never write to any of theos.StdXYZ
variables. For example: https://github.com/golang/go/blob/c4c73ce0e3a7b5c3a61678325c61b09783f76220/src/testing/testing.go#L1211is the code that writes the final "PASS" note.
A couple possible fixes might be:
testing.go
so that it reads theos.StdXYZ
variables at the start of testing, and then never reads from them again. So e.g. the above code would be updated tofmt.Fprintln(originalStdOut, "PASS")
. This protects the code from tests that tamper with the standard files.Interestingly, the issue is limited only to
-json
flag. If we rungo test
without that flag, the test will pass and no indication of anything going wrong will be given.Justification / backstory
This test case was derived from a test somebody wrote at my employer that was testing a command-line tool of some sort. It was temporarily replacing Stdout so that output could be captured and compared against expected output. The issue is that for one test, this person forgot to restore the original Stdout, and this was not noticed until I started working on introducing tools that work with the JSON output, like using
gotestsum
to convert JSON to JUnit XML and then passing that to Jenkins JUnit plug-in.... Jenkins was (surprisingly) reporting failed tests despite the test suite "passing."While replacing
os.Stdout
might arguably not the best approach to testing a command-line tool, somebody did write a test that way, and in such a scenario, I feel that the test runner should exhibit predictable behavior when presented with "interesting" tests like this one.