If you build a project using mvn verify -Dfailsafe.rerunFailingTestsCount=2, and you have @ParameterizedTest with @CsvSource, if one of the tests fail the amount of times you provided in the parameter, it will not fail the build, and this test will be considered flaky.
I believe this is a bug, because:
The build breaks as expected if the -Dfailsafe.rerunFailingTestsCount=2 is not provided
The build breaks as expected if the -Dfailsafe.rerunFailingTestsCount=2 is provided but you use @ValueSource or @MethodSource
If you build a project using
mvn verify -Dfailsafe.rerunFailingTestsCount=2
, and you have@ParameterizedTest
with@CsvSource
, if one of the tests fail the amount of times you provided in the parameter, it will not fail the build, and this test will be considered flaky.I believe this is a bug, because:
-Dfailsafe.rerunFailingTestsCount=2
is not provided-Dfailsafe.rerunFailingTestsCount=2
is provided but you use@ValueSource
or@MethodSource
Failsafe version I'm using: 3.0.0-M4
Example test file:
Note that when we try to build the project this TC is considered flaky instead of a failure: