jstemmer / go-junit-report

Convert Go test output to JUnit XML
MIT License
776 stars 224 forks source link

Failed test output can get lost with repeated and parallel test output #80

Closed dmayo3 closed 2 years ago

dmayo3 commented 5 years ago

I've been having issues with test suites that are run in parallel and repeated, the relevant test output seems to get lost or mixed up.

This minimal (hand-crafted) example seems to demonstrate the issue:

=== RUN   TestRepeat
=== RUN   TestRepeat
--- FAIL: TestRepeat (0.02 seconds)
    file_test.go:11: This is some debug logging
    file_test.go:11: Error message
    file_test.go:11: Longer
        error
        message.
--- PASS: TestRepeat (0.00s)
FAIL
FAIL      package/repeat-and-parallel    0.001s

This produces the following output with no failure message output:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
        <testsuite tests="2" failures="1" time="0.001" name="package/repeat-and-parallel">
                <properties>
                        <property name="go.version" value="go1.10.3"></property>
                </properties>
                <testcase classname="repeat-and-parallel" name="TestRepeat" time="0.000">
                        <failure message="Failed" type=""></failure>
                </testcase>
                <testcase classname="repeat-and-parallel" name="TestRepeat" time="0.000"></testcase>
        </testsuite>
</testsuites>

The problem goes away if I either give the tests unique names (as if they weren't repeated) or I change the RUN lines so as to not be interleaved (as if it weren't in parallel).

jstemmer commented 2 years ago

I'm not sure how we could attribute the output to the correct test with this specific kind of output. Fortunately, I think this is no longer an issue as Go test output uses RUN, PAUSE and CONT lines when switching between tests. Feel free to reopen if this can still be reproduced with a recent Go version (not hand-crafted).