jstemmer / go-junit-report

Convert Go test output to JUnit XML
MIT License
776 stars 224 forks source link

Benchmark failure not correctly tracked #78

Closed izgeri closed 2 years ago

izgeri commented 6 years ago

I expect that if a benchmark that I'm running fails, the XML report will note that failure. This is not necessarily the case, however.

Here is an example:

command: go test -v -bench=. -test.benchtime=10s ./bench.go

go test output:

goos: linux
goarch: amd64
Benchmark_Sample1-4            50000        227422 ns/op
--- FAIL: Benchmark_Sample10
    bench.go:43: Unexpected error
--- FAIL: Benchmark_Sample100
    bench.go:43: Unexpected error
--- FAIL: Benchmark_Sample1000
    bench.go:43: Unexpected error
--- FAIL: Benchmark_Sample10000
    bench.go:43: Unexpected error
FAIL
exit status 1
FAIL    command-line-arguments  13.986s

go-junit-report output:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
        <testsuite tests="1" failures="0" time="13.986" name="command-line-arguments">
                <properties>
                        <property name="go.version" value="go1.11"></property>
                </properties>
                <testcase classname="command-line-arguments" name="Benchmark_Sample1" time="0.000328058"></testcase>
        </testsuite>
</testsuites>

Note that the four failed benchmarks are not included in the XML output

jstemmer commented 2 years ago

I'm not able to reproduce this with the current version of go-junit-report, so I think it may have been fixed at some point.