[Spock] MaxFailures limit does not count failed specifications marked with Stepwise annotation

[AlexeyAkentyev]

Hi, I faced an issue that is quite critical for my day-to-day pipelines. I'm using Spock framework 2.x for different types of tests. While for independent tests methods 'MaxFailures' limit working just perfect it does not count failed spec annotated with Stepwise. It's quite a common case for e2e scenarios. Based on my research the issue is caused by size() implementation. https://github.com/gradle/test-retry-gradle-plugin/blob/235468201a9cd718e480c9eac4381e81eb4455f6/plugin/src/main/java/org/gradle/testretry/internal/executer/TestNames.java#L83 All scenario-level failures are sorted out as 'no failures'.

How to reproduce: I've implemented a simple test that demonstrates the issue (in the org.gradle.testretry.testframework.SpockBaseFuncTest test)

    def "handles @Stepwise tests with maxFailures limit (gradle version #gradleVersion)"() {
        given:
        buildFile << """
            test.retry {
                maxRetries = 1
                maxFailures = 1
            }
        """

        writeGroovyTestSource """
            package acme

            @spock.lang.Stepwise
            class StepwiseTests1 extends spock.lang.Specification {
                def "parentTest"() {
                    expect:
                    true
                }

                def "childTest"() {
                    expect:
                    ${flakyAssert('first')}
                }

                def "grandChildTest"() {
                    expect:
                    true
                }
            }

            @spock.lang.Stepwise
            class StepwiseTests2 extends spock.lang.Specification {
                def "parentTest"() {
                    expect:
                    true
                }

                def "childTest"() {
                    expect:
                    ${flakyAssert('second')}
                }

                def "grandChildTest"() {
                    expect:
                    true
                }
            }
        """

        when:
        def result = gradleRunner(gradleVersion).buildAndFail()

        then:
        with(result.output) {
            it.count('childTest FAILED') == 2
            it.count('childTest PASSED') == 0
            it.count('parentTest PASSED') == 2

            it.count('grandChildTest SKIPPED') == 2
            it.count('grandChildTest PASSED') == 0
        }

        where:
        gradleVersion << GRADLE_VERSIONS_UNDER_TEST
    }

Potential fix: A very simple solution is to threat tests without failed test methods as '1' instead of '0' Implementation:

    public int size() {
        return stream().mapToInt(s -> {
            int tests = s.getValue().size();
            return tests > 0 ? tests : 1;
        }).sum();
    }

I'm also fully happy with any other solution that resolves the issue. Looking forward to your reply.

P.S. I'm happy to create a pull request. It would be super great if a fix will released as a minor version after the fix.

[AlexeyAkentyev]

Hi @leonard84, Thanks a lot for reviewing and accepting the proposed idea. Is it possible to release a new version with the fix? I rely on that functionality in the CI.

[marcphilipp]

Yes, we'll release 1.5.7 in the next few days.

[leonard84]

To be more precise it was released today.