Introduce first-class support for scenario tests

[sbrannen]

Proposal

The current proposal is to introduce the following:

• @ScenarioTest: class-level annotation used to denote that a test class contains steps that make up a single scenario test.
• @Step: method-level annotation used to denote that a test method is a single step within the scenario test.

The @Step annotation will need to provide an attribute that can be used to declare the next step within the scenario test. Steps will then be ordered according to the resulting dependency graph and executed in exactly that order.

For example, a scenario test could look similar to the following:

@ScenarioTest
class WebSecurityScenarioTest {

    @Step(next = "login")
    void visitPageRequiringAuthorizationWhileNotLoggedIn() {
        // attempt to visit page which requires that a user is logged in
        // assert user is redirected to login page
    }

    @Step(next = "visitSecondPageRequiringAuthorizationWhileLoggedIn")
    void login() {
        // submit login form with valid credentials
        // assert user is redirected back to previous page requiring authorization
    }

    @Step(next = "logout")
    void visitSecondPageRequiringAuthorizationWhileLoggedIn() {
        // visit another page which requires that a user is logged in
        // assert user can access page
    }

    @Step(next = END)
    void logout() {
        // visit logout URL
        // assert user has been logged out
    }

}

Related Issues

• 13

• 419

• 607

• 884

[sbrannen]

@pgpx,

I guess I'm not sure why you would need/want each step to be a separate method (with state stored at a scenario level), and why before/after-step semantics are important (or couldn't be implemented with a simple lambda expression if needed).

Using lambda expressions and a builder pattern for implementing scenario tests is great for pure unit testing but not a good fit for complex integration and system testing.

For more complex use cases one needs the power of third-party (or self-built) extensions that tie into JUnit Jupiter's extension model and test execution lifecycle. With lambda expressions, however, that is not possible (unless you do something like this and implement #378).

[sbrannen]

I can see that complex integration tests might require a lot of setup that you don't want to repeat, but that seems to be a different concern that could maybe be handled differently (e.g. look at how Spring's integration test framework caches its own context), and those tests are not really separate steps in a single scenario.

I'm very familiar with the Spring TestContext Framework (since I'm the author of said framework 😉), so let me provide some additional context here.

It's true that one does not want to needlessly repeat expensive setup, but that's only piece of the puzzle. The other piece of the puzzle is interaction with extensions between/around steps.

Although I did not state it in this issue's description, the WebSecurityScenarioTest example I provided is actually inspired from numerous "scenario testing" use cases that I have run into with Spring Boot, Spring Security, and the Spring TestContext Framework. Thus, in real life that example should look more like the following.

@ScenarioTest
@ExtendWith(SpringExtension.class)
@SpringBootTest(webEnvironment = MOCK)
@AutoConfigureMockMvc
@Transactional
class WebSecurityScenarioTest {

    @Autowired
    MockMvc mockMvc;

    @Step(next = "login")
    void visitPageRequiringAuthorizationWhileNotLoggedIn() {
        // attempt to visit page which requires that a user is logged in
        // assert user is redirected to login page
    }

    @Step(next = "visitSecondPageRequiringAuthorizationWhileLoggedIn")
    void login() {
        // submit login form with valid credentials
        // assert user is redirected back to previous page requiring authorization
    }

    @Step(next = "logout")
    @WithMockUser(roles = "USER")
    void visitSecondPageRequiringAuthorizationWhileLoggedIn() {
        // visit another page which requires that a user is logged in
        // assert user can access page
    }

    @Step(next = END)
    @WithMockUser(roles = "USER")
    void logout() {
        // visit logout URL
        // assert user has been logged out
    }

}

Furthermore, depending on the use case, individual methods may be annotated with @Commit, @Sql, etc. in addition to the @WithMockUser annotation (and similar annotations) from Spring Security.

My point: if the steps are not methods, it is impossible for the annotations (and therefore third-party extension features) to be applied at the step level.

In summary, although the lambda expression / builder / DSL being currently discussed looks super cool and powerful, it in fact (unfortunately) has several shortcomings.

[sbrannen]

And for those interested in seeing live, working examples...

• TestNG: SimpleScenarioTestNgTests
• JUnit 4: RestEventsControllerTests
  • RestEventsControllerTests is not actually a scenario test since it's not possible to implement such a test in JUnit 4, but it could be converted to a scenario test in JUnit Jupiter (if supported).
  • In other words, ideally one should be able to implement something analogous to SimpleScenarioTestNgTests in JUnit Jupiter. 😉

[sbrannen]

Now, having said all that...

I am not saying that I am totally against some form of scenario tests based on lambda expressions and a DSL, but I think that is a separate topic that should be addressed independently.

[sormuras]

In summary, although the lambda expression / builder / DSL being currently discussed looks super cool and powerful, it in fact (unfortunately) has several shortcomings. [...] I am not saying that I am totally against some form of scenario tests based on lambda expressions and a DSL, but I think that is a separate topic that should be addressed independently.

I totally agree, @sbrannen -- that's why I put the "does not solve" disclaimer at the top of #838

All further discussion of the "fail-fast DSL dynamic tests" feature should be made at #838

[mfulton26]

One issue I see with @Step(next = "$methodName") is that there is no way to express that multiple tests follow this test but in no particular order (which would be especially useful in integration tests if concurrent execution is supported). TestNG takes the "dependsOn" route which seems to me currently to be a better route. Thoughts?

[tnimni]

is there an implementation in junit 5.0.2 for test sequence?

[marcphilipp]

@tnimni No, this issue is still open and so is #13.

[Jonarzz]

Is it known, which release will contain changes related to this issue?

[sbrannen]

It is not "known", since we cannot predict the future.

But... this issue is currently assigned to the 5.2 Backlog. Thus, the intended release is 5.2.

FYI: you can always look at the assigned milestone (see information panel to the right of any issue description on GitHub) to infer this information on your own. 😉

[mibutec]

In my company we are using a JS (i.e. Jasmine) inspired way of writing Scenarios.

@Scenario("Some succeeding scenario")
public void succeedingScenario() {
  Given("Some given condition", () -> {
    // some code
  });

  When("Some when condition", () -> {
    // some code    
  });

  Then("Some assertion" , () -> {
    // some code
  });
}

It lecks usage of JUnit default reporting, otherwise it has all the abilities of using extensions and all the abilities of java (decisions, loops, ...) to describe tests. Just as an idea.

[bobtiernay-okta]

Following up on the status of this ticket. Would be very useful to have this feature.

[sbrannen]

Following up on the status of this ticket. Would be very useful to have this feature.

This issue is currently assigned to the 5.4 backlog, and the upcoming inclusion of #13 (in 5.4 M1) will make it easier to introduce support for scenario tests.

[BlackIsTheNewBlack]

Hi,

Sorry for re-raising this issue after it has already been implemented but the "next" approach is extremely limited. Using "next" there is always only one possible order, which has to be specified completely, because there can be only one next element. This order is however arbitrary (in example configTest could also point to "buy" and "logout".

2. Removing one element from the order will force the re-assigning of the next.
3. It forces the dependants to know of each other.
4. No simple way to run a test and its dependencies, because that concept does not exist. In the example below running "logoutTest" would also force the running of "loginTest" and "buyTest", even if they are not part of the dependencies.

Example: class Config {} class Login { Config config;} class Buy { Config config;} class Logout { Config config;}

// everybody uses Config, so configTest() must succeed before running the next tests.

1: configTest(){}

2: loginTest()

3: buyTest()

4: logoutTest()

Using "next" results in: @Next("loginTest") configTest(){} @Next("buyTest") loginTest() @Next("logout") buyTest() logoutTest() // removing loginTest will also force the update of configTest

Using dependsOn results in: configTest(){} @DependsOn("configTest") loginTest() @DependsOn("configTest") buyTest() @DependsOn("configTest") logoutTest() // removing loginTest will does not force the update of configTest

[panchenko]

@RaresI In your last example, can anonymous user buy something? If not, then buyTest should depend on loginTest, logoutTest should depend on buyTest.

So, no difference - if we would like to remove buyTest - then logoutTest needs to be updated.

[BlackIsTheNewBlack]

Hello, @panchenko it seems we are actually talking about two different kind of usage models, because the terminology is not quite clear. Scenarios In a test scenario for buying, buyTest should follow loginTest, because the user has to login first. In a test scenario which verifies that auth is enforced, buyTest should not follow loginTest, because we need to check that without logging in the user is not able to checkoout. Dependencies However Config tests are necessary whenever any classes which uses configuration is used. It is critical that tests for Config are executed before tests for Buy because a failing Config class will probably make tests of dependants (Login, Buy and Logout). Requiring tests of dependencies to run before has the nice property that as a test will fail you can stop investigating all its dependencies. It is Root Cause Analysis on the cheap. My conclusion is that Scenario testing and Dependency testing are complementary, but the more common requirement is for dependency ordering. When Scenario testing is done, it should not have a fixed Next for each test method, as there are many possible paths which should be tested.

[binkley]

Hi, I'm encountering an error which directly references this issue by link. Gradle says:

* What went wrong:
Execution failed for task ':basilisk-contracts:generateContractTests'.
> Not implemented yet in JUnit5 - https://github.com/junit-team/junit5/issues/48

I'm using Spring Cloud Contract (SCC), which places all the individual contract tests into a single class, relying on an inherited test base class for setup.

I have no annotations on the test methods (SCC marks the methods with @Test from JUnit 5), and the base class of the generated test class has @ExtendsWith(SpringExtension.class), inherited as a meta annotation.

I'm surprised to see this junit failure message.

If this is the wrong place to ask about this, I'm happy to ping Spring Contract. Since the error message references here, I thought I'd ask here first.

[binkley]

Some more digging. This is definitely SCC, not JUnit, complaining:

https://github.com/spring-cloud/spring-cloud-contract/blob/8ce80e6e7fc0267a9cba1679bdf20f42ce875741/spring-cloud-contract-verifier/src/main/groovy/org/springframework/cloud/contract/verifier/config/framework/JUnit5Definition.groovy#L63

SCC believes this @TestMethodOrder(Alphanumeric.class) is unsupported. To be fair, the annotation is marked since = "5.4".

I'll talk to them.

[panchenko]

This issue looks almost completed: @TestInstance(Lifecycle.PER_CLASS) and @TestMethodOrder(MethodOrderer.OrderAnnotation.class) would work for arranging the order of the steps.

The only additional thing which is missing IMHO: skip the remaining tests after a failure.

431 looks related, but such functionality should be generic, not DynamicTest specific.

[mkobit]

@panchenko I think you are right.

Here is my first attempt I have been trying out (haven't tried with nested tests or anything complex):

import org.junit.jupiter.api.MethodOrderer
import org.junit.jupiter.api.TestInstance
import org.junit.jupiter.api.TestMethodOrder
import org.junit.jupiter.api.extension.ConditionEvaluationResult
import org.junit.jupiter.api.extension.ExecutionCondition
import org.junit.jupiter.api.extension.ExtendWith
import org.junit.jupiter.api.extension.ExtensionContext
import org.junit.jupiter.api.extension.TestExecutionExceptionHandler

// https://github.com/junit-team/junit5/issues/48 - Introduce first-class support for scenario tests
// https://github.com/junit-team/junit5/issues/431 - Introduce mechanism for terminating Dynamic Tests early

private class StepwiseExtension : ExecutionCondition, TestExecutionExceptionHandler {
  override fun handleTestExecutionException(context: ExtensionContext, throwable: Throwable) {
    val namespace = namespaceFor(context)
    val store = storeFor(context, namespace)
    store.put(StepwiseExtension::class, context.displayName)
    throw throwable
  }

  override fun evaluateExecutionCondition(context: ExtensionContext): ConditionEvaluationResult {
    val namespace = namespaceFor(context)
    val store = storeFor(context, namespace)
    val value: String? = store.get(StepwiseExtension::class, String::class.java)
    return if (value == null) {
      ConditionEvaluationResult.enabled("No test failures in stepwise tests")
    } else {
      ConditionEvaluationResult.disabled("Stepwise test disabled due to previous failure in '$value'")
    }
  }

  private fun namespaceFor(context: ExtensionContext): ExtensionContext.Namespace =
    ExtensionContext.Namespace.create(StepwiseExtension::class, context.parent)

  private fun storeFor(context: ExtensionContext, namespace: ExtensionContext.Namespace): ExtensionContext.Store =
    context.parent.get().getStore(namespace)
}

@TestInstance(TestInstance.Lifecycle.PER_CLASS)
@TestMethodOrder(MethodOrderer.OrderAnnotation::class)
@ExtendWith(StepwiseExtension::class)
@Target(AnnotationTarget.CLASS)
@Retention(AnnotationRetention.RUNTIME)
annotation class Stepwise

import org.junit.jupiter.api.Order
import org.junit.jupiter.api.Test

@Stepwise
internal class ScenarioExample {
  @Test
  @Order(1)
  internal fun first() {
    println("1")
  }

  @Test
  @Order(6)
  internal fun sixth() {
    println("6")
    throw IllegalStateException("Should be skipped")
  }

  @Test
  @Order(5)
  internal fun fifth() {
    println("5")
    throw IllegalStateException("Should be skipped")
  }

  @Test
  @Order(3)
  internal fun third() {
    println("3")
  }

  @Test
  @Order(2)
  internal fun second() {
    println("2")
  }

  @Test
  @Order(4)
  internal fun fourth() {
    println("4")
    throw IllegalArgumentException("FAILURE")
  }
}

Which fails on the fourth() test and disables fifth() and sixth().

[marcphilipp]

@mkobit Thanks for sharing, that looks promising! It should use TestWatcher instead of TestExecutionExceptionHandler to make sure it catches all test failures.

[rliesenfeld]

My two cents, since I just implemented a feature for scenario-oriented tests in my own testing tool (which I currently use with JUnit 4, but should migrate to JUnit 5 eventually).

As a user, I would prefer to simply put @ScenarioTest or @Stepwise on my test classes, and use regular @Test methods for the scenario steps. Having to explicitly specify a numerical order, or worse, by test name, is just too cumbersome.

Test (step) ordering should respect the textual order I write the tests in the test class. This can be implemented without much difficulty using ASM. (I just did it, hacking JUnit 4's TestClass, but probably won't release this in my tool.) The Java classfile normally preserves textual order, but even if it didn't, textual ordering could still be guaranteed by reading the "LineNumberTable" classfile attribute with an ASM MethodVisitor.

[sbrannen]

Test (step) ordering should respect the textual order I write the tests in the test class. This can be implemented without much difficulty using ASM. (I just did it, hacking JUnit 4's TestClass, but probably won't release this in my tool.) The Java classfile normally preserves textual order, but even if it didn't, textual ordering could still be guaranteed by reading the "LineNumberTable" classfile attribute with an ASM MethodVisitor.

That's related to the discussion here: https://github.com/junit-team/junit5/issues/1919#issuecomment-499952534

[vab2048]

Is it a goal of this feature to support mapping from a Gherkin feature file to the relevant scenario classes? Or any sort of first class integration with cucumber?

[marcphilipp]

Is it a goal of this feature to support mapping from a Gherkin feature file to the relevant scenario classes?

No, this is about providing sth. similar to Spock's @Stepwise in JUnit Jupiter.

Or any sort of first class integration with cucumber?

AFAIK Cucumber already provides a custom TestEngine so integration should no longer be an issue or am I missing sth.?

[marcin-chwedczuk]

Extension provided by @mkobit works fine for me. Here is Java version:

package pl.marcinchwedczuk.javafx.validation.demo.utils;

import org.junit.jupiter.api.extension.ConditionEvaluationResult;
import org.junit.jupiter.api.extension.ExecutionCondition;
import org.junit.jupiter.api.extension.ExtensionContext;
import org.junit.jupiter.api.extension.TestExecutionExceptionHandler;

// https://github.com/junit-team/junit5/issues/48 - Introduce first-class support for scenario tests

class StopOnFirstFailureExtension implements ExecutionCondition, TestExecutionExceptionHandler {
    @Override
    public void handleTestExecutionException(ExtensionContext context, Throwable throwable) throws Throwable {
        try {
            ExtensionContext.Namespace namespace = namespaceFor(context);
            ExtensionContext.Store store = storeFor(context, namespace);
            store.put(StopOnFirstFailureExtension.class, context.getDisplayName());
        } catch (Exception e) {
            e.printStackTrace();
        }

        throw throwable;
    }

    @Override
    public ConditionEvaluationResult evaluateExecutionCondition(ExtensionContext context) {
        var namespace = namespaceFor(context);
        var store = storeFor(context, namespace);

        var value = store.get(StopOnFirstFailureExtension.class, String.class);
        if (value == null) {
            return ConditionEvaluationResult.enabled("No test failures in stepwise tests");
        } else {
            return ConditionEvaluationResult.disabled(
                    "Stepwise test disabled due to previous failure in '" + value + "'");
        }
    }

    private ExtensionContext.Namespace namespaceFor(ExtensionContext context) {
        return ExtensionContext.Namespace.create(StopOnFirstFailureExtension.class, context.getParent());
    }

    private ExtensionContext.Store storeFor(ExtensionContext context, ExtensionContext.Namespace namespace) {
        return context.getParent().get().getStore(namespace);
    }
}

@TestInstance(TestInstance.Lifecycle.PER_CLASS)
@ExtendWith(StopOnFirstFailureExtension.class)
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
public @interface StopOnFirstFailure { }

[marcphilipp]

@nipafx Could this be a feature for JUnit Pioneer?

[jbduncan]

For anyone stumbling upon this, I've found the Test Robot Pattern a great way of expressing "steps" or "scenarios" inside a test in a readable manner. My current team uses it in an Android project to great effect, and the pattern also applies to things like Selenium tests.

[nipafx]

@nipafx Could this be a feature for JUnit Pioneer?

Yes. 😁 I assume that means that Jupiter will not implement this (for now)?

[marcphilipp]

It seems there's a relatively simple way to achieve this using already existing extensible mechanisms so I would tend not to. 🙂

[Saljack]

I do not understand why don't you want to add it directly to JUnit? Or do you want to test it in JUnit Pioneer and then move it to JUnit? Or are there another reasons? Because I think this is one of the most wanted feature.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. Thank you for your contribution.

[igilbert]

Note that the workaround offered by StopOnFirstFailureExtension is quite different with inter-test dependency. The latter should work like this: when running a step3 test, its dependence tests would run as well. this happens when debugging step3 test, while the workaround would not run step1-2 tests at all.

[marcphilipp]

For anyone landing on this page, looking for a complete example of a StopOnFirstFailureExtension, please see https://github.com/junit-team/junit5/issues/3099#issuecomment-1333545198.