spockframework / spock

The Enterprise-ready testing and specification framework.
https://spockframework.org
Apache License 2.0
3.56k stars 470 forks source link

Test failures coming from JUnit4 rules are incorrectly attributed to subsequent iterations of unrolled features #1268

Open rieske opened 3 years ago

rieske commented 3 years ago

Issue description

When JUnit4 rule throws while handling an unrolled feature iteration, the test failure is incorrectly attributed to the subsequent iteration. The outcome looks identical to https://github.com/spockframework/spock/issues/1267 and could point to the same root cause.

How to reproduce

import org.junit.Rule
import org.junit.rules.TestRule
import org.junit.runner.Description
import org.junit.runners.model.Statement
import spock.lang.Specification

class RuleSpec extends Specification {

    @Rule
    public SomeRule rule = new SomeRule()

    def "feature with rule"() {
        expect:
        true

        where:
        foo << ['a', 'b', 'c', 'd']
    }
}

class SomeRule implements TestRule {
    @Override
    Statement apply(Statement base, Description description) {
        return new Statement() {
            @Override
            void evaluate() throws Throwable {
                base.evaluate()
                throw new RuntimeException("failure in rule")
            }
        }
    }
}

Executing the above test will report the failures like this: image

Additional Environment information

Build-tool dependencies used

testImplementation("org.spockframework:spock-core:2.0-M4-groovy-3.0")
testImplementation("org.spockframework:spock-junit4:2.0-M4-groovy-3.0")
leonard84 commented 3 years ago

Rule support is made on a best-effort basis, as they don't perfectly match with Spocks concepts, but works well enough most of the time. However, there might be cases - such as this one - where it doesn't perfectly work, I'd suggest to create a native Spock extension if possible.