Fail in the TestMain not recognized

[taraspos]

Hey. We have a file tests.go which does some initialization in the func TestMain(m *testing.M).

If initialization fails go-junit-report does not recognize it as a fail (probably because test name is missing in the output, since there are no test name yet). Example:

?       my_pacakge/file1    [no test files]
?       my_pacakge/file2    [no test files]
time="2018-03-28T12:47:03Z" level=fatal msg="cannot connect to mysql: ..."
FAIL    my_packege/tests    0.156s
=== RUN   Test1
--- PASS: Test1 (0.00s)
=== RUN   Test2
--- PASS: Test2 (0.00s)

How we start tests: go test -v ./... 2>&1 | tee /dev/stderr | go-junit-report -set-exit-code > test-reports/report.xml

Since fail was not recognized commands exit with 0 exit code.

We figured out the workaround, running: set -o pipefail; go test -v ./... 2>&1 | tee /dev/stderr | go-junit-report -set-exit-code > test-reports/report.xml this way command exits with non-0 exitcode. But would be nice to have it natively handled with proper report.

Thanks

[jstemmer]

You didn't include the generated report.xml, what did it look like?

Running your output through go-junit-report results in an XML that reports my_package/tests as having a failure (it even includes the correct failure message) and the exit code is 1.

$ cat /tmp/output
?       my_pacakge/file1        [no test files]
?       my_pacakge/file2        [no test files]
time="2018-03-28T12:47:03Z" level=fatal msg="cannot connect to mysql: ..."
FAIL    my_packege/tests        0.156s
=== RUN   Test1
--- PASS: Test1 (0.00s)
=== RUN   Test2
--- PASS: Test2 (0.00s)

$ cat /tmp/output | go-junit-report --set-exit-code ; echo $?
<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
        <testsuite tests="1" failures="1" time="0.156" name="my_packege/tests">
                <properties>
                        <property name="go.version" value="go1.10"></property>
                </properties>
                <testcase classname="tests" name="Failure" time="0.000">
                        <failure message="Failed" type="">time=&#34;2018-03-28T12:47:03Z&#34; level=fatal msg=&#34;cannot connect to mysql: ...&#34;</failure>
                </testcase>
        </testsuite>
        <testsuite tests="2" failures="0" time="0.000" name="">
                <properties>
                        <property name="go.version" value="go1.10"></property>
                </properties>
                <testcase classname="" name="Test1" time="0.000"></testcase>
                <testcase classname="" name="Test2" time="0.000"></testcase>
        </testsuite>
</testsuites>
1

Are you using the latest version of go-junit-report? And is the output you pasted accurate?

[taraspos]

@jstemmer hey, sorry, got a bit distracted and forgot about this issue. I will try to reproduce it next week and share the results with you.

[taraspos]

@jstemmer hey, I was able to reproduce the issue. The example I posted above worked fine, because I shortened, and apparently removed the pattern :(

So here is the example that not being detected as problem:

=== RUN   Test0
--- PASS: Test0 (0.00s)
PASS
?       my_pacakge/file1    [no test files]
?       my_pacakge/file2    [no test files]
time="2018-03-28T12:47:03Z" level=fatal msg="cannot connect to mysql: ..."
FAIL    my_packege/tests    0.156s
=== RUN   Test1
--- PASS: Test1 (0.00s)
=== RUN   Test2
--- PASS: Test2 (0.00s)

It is completely the same, just added:

=== RUN   Test0
--- PASS: Test0 (0.00s)
PASS

at the beginning

[jstemmer]

I see, this is indeed not detected as a problem currently. It looks like the output you posted does not contain a summary line (example), and I suspect that is why the failure is not captured at the moment. Any idea why this is missing?

I haven't been able to reproduce this myself, can you post a minimal example that produces this output when running go test -v ./...?

[taraspos]

Hey, there is summary actually, I guess I just forgot it while was sanitizing the output. Here is one more example:

=== RUN   Test0
--- PASS: Test0 (0.00s)
PASS
ok      test/0  0.030s
?       test/1  [no test files]
?       test/2  [no test files]
time="2018-04-23T19:01:04+03:00" level=fatal msg="cannot connect to mysql: dial tcp: lookup mysql: no such host"
FAIL    test/3  0.061s
=== RUN   Test1
--- PASS: Test1 (0.00s)
=== RUN   Test2
--- PASS: Test2 (0.00s)
=== RUN   Test3
--- PASS: Test3 (0.00s)
=== RUN   Test4
--- PASS: Test4 (0.00s)
PASS
ok      test/3  0.060s
?       test/4  [no test files]
?       test/5  [no test files]

Also, I will try to prepare the project with the failing tests for you.

[taraspos]

@jstemmer here you go: https://github.com/Trane9991/go-junit-report-issue

[jstemmer]

Thanks for the example, I can reproduce it now. This is definitely a problem and I can see where it goes wrong. Once I find some time I'll see if I can fix it.

[wknapik]

Just ran into this. Output with prints from the tests stripped out:

--- PASS: foo (0.86s)
PASS
FAIL    bar 222.957s

go-junit-report reported 0 failures and 0 errors.

The way I understand it is that os.exit(non-zero) was called at the end of the test suite and that caused a failure on the module level (FAIL bar), but there was no failing test, so go-junit-report didn't report failing tests.

I guess a solution would be to say <testsuite errors="1"> ?

Do you have an ETA on the fix ?;]

[jstemmer]

This has been fixed in v2.0.0-beta1.