search

TNG / junit-dataprovider

A TestNG like dataprovider runner for JUnit with many additional features
Apache License 2.0
246 stars 164 forks source link

rerunFailingTestsCount + @DataProvider = failed tests are recognized as flakes #138

Closed camproto closed 6 months ago

camproto commented 1 year ago

Overview

Bug report.

The combination of rerunFailingTestsCount=3 of the maven surefire plugin with the TestNG junit dataprovider did not fail the test, instead it's just marked as flaky, even the count is reached.

There was an issue on the surefire plugin which discusses exactly this problem:
SUREFIRE-1228 (contains a test case as well).

But it seems to me, that the dataprovider is not fully working together with the surefire rerunFailingTestsCount option.

ACTUAL: The maven log ends just flaky (full report in attachment) [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1

EXPECTED: The test goes into failure, because even after rerun 3 times the test is not passed.

Versions tested maven-surefire-plugin is latest 3.0.0-M7 com.tngtech.java:junit-dataprovider:1.13.1

I tried even your latest version com.tngtech.junit.dataprovider:junit4-dataprovider:2.9, but still the same maven result: just marked as flaky, but not goes to failure.

Deliverables

maven-log-just-flaky.txt

aaschmid commented 1 year ago

@camproto Thanks for the feedback. I am unsure about your current setup. Do you use TestNG? Does everything work as expected without the rerunFailingTestsCount flag?

camproto commented 1 year ago

Thanks @aaschmid for noting it. We use JGiven in combination with the TNG junit-dataprovider. I can confirm, that without the rerunFailingTestsCountflag everything works as expected. We face some flaky tests and tried this flag to prevent manually re-start our build jobs.

aaschmid commented 1 year ago

Ok, understood. Could you easily provide reproducer - this would help a lot.

camproto commented 1 year ago

Please refer to SUREFIRE-1228 for the test case.

camproto commented 1 year ago

One additional information which might be related. The surefire plugin is running via org.junit.vintage:junit-vintage-engine:5.8.2 within the junit jupiter container (from SpringBoot 2.6.2).

camproto commented 1 year ago

Another point I'm not sure you have an influence on. Our Jenkins job is not getting yellow, if there were flaky tests. All are getting just green. Yellow is expecting to make the flakiness visible.

aaschmid commented 1 year ago

Another point I'm not sure you have an influence on. Our Jenkins job is not getting yellow, if there were flaky tests. All are getting just green. Yellow is expecting to make the flakiness visible.

Maybe a consequential error of this or have you checked without a dataprovider as well?

The test result in Jenkins is usually interpreted by some (junit) plugin which marks the build properly. JUnit vintage is responsible for providing the correct information in a result file (likely some XML depending on the configuration). Maven surefire plugin is responsible for the exit code - if non-zero Jenkins assume as build failure.

camproto commented 1 year ago

Thank you for your explanation! I did not checked it without the dataprovider. But you might be right, it seems to be just a consequential error.

aaschmid commented 1 year ago

@camproto: Can you check it without the dataprovider in order to preclude an error with Maven?

aaschmid commented 6 months ago

Closing issue due to inactivity.