search

gotestyourself / gotestsum

'go test' runner with output optimized for humans, JUnit XML for CI integration, and a summary of the test results.
Apache License 2.0
1.99k stars 118 forks source link

`--junitfile` XML report contains all test runs if using `--rerun-fails` #384

Closed valdisrigdon closed 6 months ago

valdisrigdon commented 7 months ago

We are (unfortunately) using --rerun-fails to re-run a few flaky tests. When the subsequent re-run of the test passes, we end up with an exit code of 0 but the junit report parsing still indicates that tests failed.

For my use case, the point of using --rerun-fails is to mask the flaky test. --rerun-fails-report seems to be added to support capturing those failures.

Would it be possible to stop outputting the earlier failures to the junit file?

For example, if TestA failed in the first run but passed in the second attempt, the junit XML would have 2 entries for TestA, with the first entry being the failure and the second one being the successful run.

dnephin commented 6 months ago

Hello! This is by design. Hiding the failure could be harmful. I believe this is a duplicate of #281. Would you be ok to close this and continue the discussion in that issue?

What problems does the output cause? Maybe we can address those problems in some other way?

I should also mention that https://github.com/golang/go/issues/62244 is an accepted proposal to add retrying to the stdlib. When that's implemented gotestsum will likely change to use that support more directly. In that proposal I believe all failures are output, but that may give us a better way to output the flaky failures to make it more clear that they were retried.

valdisrigdon commented 6 months ago

Yes, I'll close this as it's a duplicate of #281