search

jstemmer / go-junit-report

Convert Go test output to JUnit XML
MIT License
763 stars 222 forks source link

Possibility of convert new fuzzing tests output to junit format. #129

Open LorenzoWF opened 2 years ago

LorenzoWF commented 2 years ago

On the new golang version 1.18 we have a new way to test, called fuzzing tests, and this return a different output. Is it possible convert this new output to junit format?

jstemmer commented 2 years ago

I haven't used the fuzzing yet in Go 1.18, but a quick test shows the output is somewhat similar so it might work already. Have you tried it? If you have some fuzz output that is not handled well, feel free to post it here.

LorenzoWF commented 2 years ago

Hey, thanks for you reply. Yes, i tested, and the fuzzing format it's a little different. You can using this example, but i generated the following output in a little personal project using this command:

Go Command:

go test -mod=vendor -failfast ./... -bench=. -fuzz=Fuzz -fuzztime 10s -v > $(TEST_REPORT)/test.report

Go Output:

=== RUN   TestNew
=== RUN   TestNew/All_equal
=== RUN   TestNew/Only_type_equal
--- PASS: TestNew (0.00s)
    --- PASS: TestNew/All_equal (0.00s)
    --- PASS: TestNew/Only_type_equal (0.00s)
=== RUN   TestWrap
=== RUN   TestWrap/All_equal
--- PASS: TestWrap (0.00s)
    --- PASS: TestWrap/All_equal (0.00s)
=== RUN   TestWrapf
=== RUN   TestWrapf/All_equal
--- PASS: TestWrapf (0.00s)
    --- PASS: TestWrapf/All_equal (0.00s)
=== RUN   TestUnwrap
=== RUN   TestUnwrap/Simple_Unwrap
--- PASS: TestUnwrap (0.00s)
    --- PASS: TestUnwrap/Simple_Unwrap (0.00s)
=== RUN   TestCause
=== RUN   TestCause/Simple_Cause
--- PASS: TestCause (0.00s)
    --- PASS: TestCause/Simple_Cause (0.00s)
=== RUN   TestServiceError_New
=== RUN   TestServiceError_New/Success_new
--- PASS: TestServiceError_New (0.00s)
    --- PASS: TestServiceError_New/Success_new (0.00s)
=== RUN   TestServiceError_Wrap
=== RUN   TestServiceError_Wrap/Success_new
--- PASS: TestServiceError_Wrap (0.00s)
    --- PASS: TestServiceError_Wrap/Success_new (0.00s)
=== RUN   TestServiceError_Wrapf
=== RUN   TestServiceError_Wrapf/Success_new
--- PASS: TestServiceError_Wrapf (0.00s)
    --- PASS: TestServiceError_Wrapf/Success_new (0.00s)
goos: linux
goarch: amd64
pkg: git.worten.net/fcd/packages/error
cpu: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
BenchmarkWrapf
BenchmarkWrapf-8     1000000          1054 ns/op
=== FUZZ  FuzzUnwrap
fuzz: elapsed: 0s, gathering baseline coverage: 0/2 completed
fuzz: elapsed: 0s, gathering baseline coverage: 2/2 completed, now fuzzing with 8 workers
fuzz: elapsed: 3s, execs: 394675 (131535/sec), new interesting: 0 (total: 2)
fuzz: elapsed: 6s, execs: 716357 (107244/sec), new interesting: 0 (total: 2)
fuzz: elapsed: 9s, execs: 927277 (70285/sec), new interesting: 0 (total: 2)
fuzz: elapsed: 10s, execs: 995453 (63186/sec), new interesting: 0 (total: 2)
--- PASS: FuzzUnwrap (10.08s)
PASS
ok      git.worten.net/fcd/packages/error   11.154s

The XML generated with go-junit-report:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
    <testsuite tests="18" failures="0" time="11.154" name="git.worten.net/fcd/packages/error">
        <properties>
            <property name="go.version" value="go1.18"></property>
        </properties>
        <testcase classname="error" name="TestNew" time="0.000"></testcase>
        <testcase classname="error" name="TestNew/All_equal" time="0.000"></testcase>
        <testcase classname="error" name="TestNew/Only_type_equal" time="0.000"></testcase>
        <testcase classname="error" name="TestWrap" time="0.000"></testcase>
        <testcase classname="error" name="TestWrap/All_equal" time="0.000"></testcase>
        <testcase classname="error" name="TestWrapf" time="0.000"></testcase>
        <testcase classname="error" name="TestWrapf/All_equal" time="0.000"></testcase>
        <testcase classname="error" name="TestUnwrap" time="0.000"></testcase>
        <testcase classname="error" name="TestUnwrap/Simple_Unwrap" time="0.000"></testcase>
        <testcase classname="error" name="TestCause" time="0.000"></testcase>
        <testcase classname="error" name="TestCause/Simple_Cause" time="0.000"></testcase>
        <testcase classname="error" name="TestServiceError_New" time="0.000"></testcase>
        <testcase classname="error" name="TestServiceError_New/Success_new" time="0.000"></testcase>
        <testcase classname="error" name="TestServiceError_Wrap" time="0.000"></testcase>
        <testcase classname="error" name="TestServiceError_Wrap/Success_new" time="0.000"></testcase>
        <testcase classname="error" name="TestServiceError_Wrapf" time="0.000"></testcase>
        <testcase classname="error" name="TestServiceError_Wrapf/Success_new" time="0.000"></testcase>
        <testcase classname="error" name="BenchmarkWrapf" time="0.000001054"></testcase>
    </testsuite>
</testsuites>

Note that it does not include the fuzzing tests of my golang output, but don't worry, i will send a PR in the next days :)

jstemmer commented 2 years ago

Thanks, that's very helpful. It looks like we should be able to support this with a few small changes, and in fact it looks like the v2 branch already outputs something that seems usable:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="19">
        <testsuite name="git.worten.net/fcd/packages/error" tests="19" failures="0" errors="0" hostname="local" time="11.154" timestamp="2022-04-05T21:23:09+01:00">
                <testcase name="TestNew" classname="git.worten.net/fcd/packages/error" time="0.000"></testcase>
                <testcase name="TestNew/All_equal" classname="git.worten.net/fcd/packages/error" time="0.000"></testcase>
                <testcase name="TestNew/Only_type_equal" classname="git.worten.net/fcd/packages/error" time="0.000"></testcase>
                <testcase name="TestWrap" classname="git.worten.net/fcd/packages/error" time="0.000"></testcase>
                <testcase name="TestWrap/All_equal" classname="git.worten.net/fcd/packages/error" time="0.000"></testcase>
                <testcase name="TestWrapf" classname="git.worten.net/fcd/packages/error" time="0.000"></testcase>
                <testcase name="TestWrapf/All_equal" classname="git.worten.net/fcd/packages/error" time="0.000"></testcase>
                <testcase name="TestUnwrap" classname="git.worten.net/fcd/packages/error" time="0.000"></testcase>
                <testcase name="TestUnwrap/Simple_Unwrap" classname="git.worten.net/fcd/packages/error" time="0.000"></testcase>
                <testcase name="TestCause" classname="git.worten.net/fcd/packages/error" time="0.000"></testcase>
                <testcase name="TestCause/Simple_Cause" classname="git.worten.net/fcd/packages/error" time="0.000"></testcase>
                <testcase name="TestServiceError_New" classname="git.worten.net/fcd/packages/error" time="0.000"></testcase>
                <testcase name="TestServiceError_New/Success_new" classname="git.worten.net/fcd/packages/error" time="0.000"></testcase>
                <testcase name="TestServiceError_Wrap" classname="git.worten.net/fcd/packages/error" time="0.000"></testcase>
                <testcase name="TestServiceError_Wrap/Success_new" classname="git.worten.net/fcd/packages/error" time="0.000"></testcase>
                <testcase name="TestServiceError_Wrapf" classname="git.worten.net/fcd/packages/error" time="0.000"></testcase>
                <testcase name="TestServiceError_Wrapf/Success_new" classname="git.worten.net/fcd/packages/error" time="0.000"></testcase>
                <testcase name="FuzzUnwrap" classname="git.worten.net/fcd/packages/error" time="10.080"></testcase>
                <testcase name="BenchmarkWrapf" classname="git.worten.net/fcd/packages/error" time="1.054000000"></testcase>
        </testsuite>
</testsuites>

I'd hold off on sending a PR at the moment. I don't intend to make further changes to v1 (currently in the master branch). I'm working on a v2 (in the v2 branch), but still planning to do some refactoring before I merge it to master and start accepting contributions for that version.