search

summerwind / h2spec

A conformance testing tool for HTTP/2 implementation.
MIT License
661 stars 73 forks source link

JUnit fails to report test errors: error is *errors.errorString, not *spec.TestError #63

Closed Karm closed 7 years ago

Karm commented 7 years ago

JUnit reporter can handle test Pass and test Failure but it cannot handle a test Error, e.g. server timeout, misconfiguration, arbitrary test implementation error.

Example

/home/karm/go/src/github.com/Karm/h2spec/bin/h2spec -h 192.168.122.204 -k --junit-report junit.report -p 3443 -t -v

Actual result

  5.4.1. Connection Error Handling
    × 1: Sends invalid frame for connection close
    Error: <nil>

panic: interface conversion: error is *errors.errorString, not *spec.TestError

goroutine 1 [running]:
panic(0x6faaa0, 0xc4202a8980)
    /opt/go/src/runtime/panic.go:500 +0x1a1
github.com/summerwind/h2spec/reporter.convertJUnitReport(0xc42006f490, 0x2, 0x2, 0x0, 0x0, 0x0)
    /home/karm/go/src/github.com/Karm/h2spec/reporter/junit_report.go:102 +0xb36
github.com/summerwind/h2spec/reporter.convertJUnitReport(0xc42010c9a0, 0x3, 0x4, 0x0, 0x0, 0x0)
    /home/karm/go/src/github.com/Karm/h2spec/reporter/junit_report.go:69 +0x1a5
github.com/summerwind/h2spec/reporter.convertJUnitReport(0xc42004bba0, 0x1, 0x1, 0x0, 0x0, 0x3)
    /home/karm/go/src/github.com/Karm/h2spec/reporter/junit_report.go:69 +0x1a5
github.com/summerwind/h2spec/reporter.JUnitReport(0xc42004bba0, 0x1, 0x1, 0x7ffd6e1e563e, 0x1, 0xc4202bae60, 0x1b)
    /home/karm/go/src/github.com/Karm/h2spec/reporter/junit_report.go:51 +0x5d
github.com/summerwind/h2spec.Run(0xc42006a7e0, 0xc42006a7e0, 0x9)
    /home/karm/go/src/github.com/Karm/h2spec/h2spec.go:37 +0x345
main.run(0xc420071440, 0xc4200982d0, 0x0, 0x9, 0x0, 0x0)
    /home/hudson/go/src/github.com/Karm/h2spec/cmd/h2spec/h2spec.go:137 +0x49e
github.com/spf13/cobra.(*Command).execute(0xc420071440, 0xc420082010, 0x9, 0x9, 0xc420071440, 0xc420082010)
    /home/hudson/go/src/github.com/spf13/cobra/command.go:632 +0x23e
github.com/spf13/cobra.(*Command).ExecuteC(0xc420071440, 0x752cbf, 0x1a, 0xc42006f3fa)
    /home/hudson/go/src/github.com/spf13/cobra/command.go:722 +0x367
github.com/spf13/cobra.(*Command).Execute(0xc420071440, 0x74b5f2, 0x4)
    /home/hudson/go/src/github.com/spf13/cobra/command.go:681 +0x2b
main.main()
    /home/hudson/go/src/github.com/Karm/h2spec/cmd/h2spec/h2spec.go:44 +0x454

Expected

Correct JUnit report saying that such and such test ended up with an Error, not Failure. e.g. Timeout:

<testsuite name="5.1.2. Stream Concurrency" package="http2/5.1.2" id="5.1.2" tests="1" skipped="0" failures="1" errors="0">
  <testcase package="http2/5.1.2" classname="Sends HEADERS frames that causes their advertised concurrent stream limit to be exceeded" time="2.0001">
    <failure>Test in Error: Timeout</failure>
  </testcase>
</testsuite>
summerwind commented 7 years ago

Thank you for using the v2 branch! The v2 branch is currently under development and this is a bug. The fixes are landed in ce1dece4c6e4ae24a6a83bf77a084a34dd6b9e3f and 9ab46960ed7b45d6f44930591996e680e68d97bc.