Report is not complete when some flaky tests are rerun with surefire

[vduhautois]

My "issue" here is we run our tests with maven (surefire) and we retry the failed tests (with the surefire.rerunFailingTestsCount option).

As a result, in the TEST-junit-jupiter.xml report file, I have only the rerun tests. I try to investigate a bit about the behaviour and my conclusion is:

• for the first run, I have a complete report file
• when surefire rerun failed tests, the report file is overwritten with this second execution, and only the rerun tests are present here.

We do not retrieve the behaviour of surefire report, which adds a flakyFailure inside a testCase, and generates a unique file with all executions.

I know this behaviour is dedicated to surefire and not to junit, but I was wondering if this is something that can be reproduced in the xray-junit-extensions?

Thanks

[bitcoder]

I'm not aware of that behavior details. How does junit5 and surefire work in that scenario, without having xray-junit-extensions? I would expect it behaves in the same away as this implementation is mostly a evolution on top of junit5 code. If so, then probably this needs to be addressed elsewhere.. junit5? Surefire? I don't know. Can you make a sample code/repo showcasing that behavior?

[vduhautois]

Sure, I will do that quickly

[vduhautois]

@bitcoder You can reproduce the behavior launching tests in this repo: https://github.com/vduhautois/xrayJunitExtensionFlakyTestIssue

I took some screenshot anyway to show the difference between surefire vs enhanced report here. PS: as said in the README of the created repo, I run the tests with the following command: mvn test -Dsurefire.rerunFailingTestsCount=2

Here is tests result:

Here is surefire report for the class, as you can see, 2 test cases including 1 with a flaky failure:

And here is the enhanced report, with only the flaky test:

[bitcoder]

Thanks @vduhautois for sharing. It seems that your junit xml report is being generated by surefire. I wasnt aware of those custom XML elements; I also don't know if any tool is able of process them (I've seen many junit xml and several schemas and this never came across). anyway, I'm not sure how surefire runs the tests. The current implementation I've is based on the legacy XML writer from the junit5 team.. so mostly for sure their default/legacy report also doesnt support this. Not sure of how to proceed as this is not a surefire extension.. ideas?

[vduhautois]

In fact, I'm not sure this behavior exists in junit unless you write an Extension (Rule in junit4). Surefire might have a specific listener that rerun failed tests (when dedicated option is set) and write the report at the end.

So i'm not really surprised your current implementation do not reproduce that. Maybe we should dig on how surefire deal with this ?

[bitcoder]

Help is appreciated @vduhautois :) I briefly searched for the code where that flakyFailure is added but I didnt come to a clear conclusion. Besides, here there's a question that we need to think first of all. That behavior is implemented by the surefire runner right? which is not what this extension assumes; this one is built on top of junit5.

[vduhautois]

https://github.com/apache/maven-surefire/blob/79db90338fb474f91c76991388a35bc412ee1d46/surefire-providers/surefire-junit47/src/main/java/org/apache/maven/surefire/junitcore/JUnitCoreProvider.java

Surefire has its own implementation and listener to deal with test execution, report, and manage failed tests to rerun. In fact, if your extension is built on top of junit5, manage such behavior is not what you want. It would mean a dedicated implementation based on surefire reporting and not junit reporting

[bitcoder]

Hmmm if I understood correctly, you mean that we would need to generate the XML report at surefire level using some sort of extension/plugin mechanism of surefire? Have you seen any examples of that?

[vduhautois]

I've never seen anything like that, your extension is the first one I've ever used :) And there is a thing: surefire generate 1 report file per test class whereas you generate a unique report for all tests. So here and now, I really have no idea of how it can be done (I have no experience on junit report extension)

[bitcoder]

I also have no idea how to handle this. Besides, it's a format that I've not seen so far. Maybe the workaround would be to do a merge of junit xml reports... not sure..

[vduhautois]

I've just thinked about one thing... If we had the ability to configure the name of the generated report (by adding timestamp for example), we will be able to have one report per run. Like that, the report generated by the first run won't be overwritten by the 2nd run when failed tests are rerun.

Do you think it is doable ?

[bitcoder]

Well, that seams a reasonable solution

[bitcoder]

made a proof-of-concept implementation, on #13 . Please check the branch custom_report_name.

[vduhautois]

I checkout the branch and test it locally. It will do the trick, that sounds great!

I still have a suggestion regarding the output path of the report(s): -> As your project is a maven one and deploy as a maven dependency, should not we write the reports somewhere in the ./target directory instead of somewhere at the root of the project?

[bitcoder]

I still have a suggestion regarding the output path of the report(s): -> As your project is a maven one and deploy as a maven dependency, should not we write the reports somewhere in the ./target directory instead of somewhere at the root of the project?

I think that can make sense but it's best to create an issue for that so we can track it independently. thanks!

[bitcoder]

@vduhautois I've updated the PR, can you please check that branch to see if it meets all your needs? More info on the PR

[bitcoder]

closing as it was addressed indirectly on PR #13 ; it doesn't do exactly the same but seems a viable approach/workaround

[ZescherW]

Hmmm if I understood correctly, you mean that we would need to generate the XML report at surefire level using some sort of extension/plugin mechanism of surefire? Have you seen any examples of that?

I just come across this after opening an issue similar to this. Allure Framework is able to identify flaky tests when surefire.rerunFailingTestsCount parameter is set.