Rerunning failing tests with JUnit @ParamaterizedTest @CsvSource does not break the build

[gr-ganzaroli]

If you build a project using mvn verify -Dfailsafe.rerunFailingTestsCount=2, and you have @ParameterizedTest with @CsvSource, if one of the tests fail the amount of times you provided in the parameter, it will not fail the build, and this test will be considered flaky.

I believe this is a bug, because:

• The build breaks as expected if the -Dfailsafe.rerunFailingTestsCount=2 is not provided
• The build breaks as expected if the -Dfailsafe.rerunFailingTestsCount=2 is provided but you use @ValueSource or @MethodSource

Failsafe version I'm using: 3.0.0-M4

Example test file:

public class ParameterizedTests {

    static Stream<Arguments> methodSource() {
        return Stream.of(
                Arguments.of("false"),
                Arguments.of("false"),
                Arguments.of("false"),
                Arguments.of("false"),
                Arguments.of("false"),
                Arguments.of("false"),
                Arguments.of("false"),
                Arguments.of("true"),
                Arguments.of("false"),
                Arguments.of("false")
        );
    }

    @DisplayName("testingStringArguments")
    @ParameterizedTest
//    @ValueSource(strings = {"false", "false", "false", "false", "false", "false", "false", "true", "false", "false"})
    @ValueSource(strings = {"true"})
    public void parameterizedString(final String shouldPass) {
        switch (shouldPass) {
            case "false":
                assertTrue(false);
            case "true":
                assertTrue(true);
        }
    }

    @DisplayName("testingMethodArguments")
    @ParameterizedTest
//    @MethodSource("methodSource")
    @MethodSource("passMethodSource")
    public void parameterizedMethod(final String shouldPass) {
        switch (shouldPass) {
            case "false":
                assertTrue(false);
            case "true":
                assertTrue(true);
        }
    }

    @DisplayName("testingCsvArguments")
    @ParameterizedTest
    @CsvSource(delimiter = ';', value = {
            "false; qwer",
            "false; qwer",
            "false; qwer",
            "false; qwer",
            "false; qwer",
            "false; qwer",
            "false; qwer",
            "true; qwer",
            "false; qwer",
            "false; qwer"
    })
//    @CsvSource(delimiter = ';', value = {
//            "true; qwer"
//    })
    public void parameterizedCsv(final String shouldPass, final String uselessParameter) {
        switch (shouldPass) {
            case "false":
                assertTrue(false);
            case "true":
                assertTrue(true);
        }
    }
}

Note that when we try to build the project this TC is considered flaky instead of a failure:

[gr-ganzaroli]

Updating to version 3.0.0-M7 fixed the issue.