Nested Steps Without SerenityReporterParallel plugin Incorrectly Pass Tests

[mkonir0]

What happened?

When using nested steps in Serenity 4 with Cucumber and simultaneously not using the io.cucumber.core.plugin.SerenityReporterParallel plugin, tests that are expected to fail are incorrectly marked as passed. This issue started occurring only after upgrading from Serenity 3 to Serenity 4 ( it worked fine before). I would also like to mention that we do not want to run steps in parallel, so using the SerenityReporterParallel plugin is not an option for us.

What did you expect to happen?

since nested step contains failing assertion tests should end up as failure.

Serenity BDD version

4.0.15 (same issue is in 4.0.21)

JDK version

17

Execution environment

4.0.15 5.1.0 1.10.0 7.14.0 ### How to reproduce the bug. To demonstrate the issue, I have created a fork of the ` serenity-cucumber-starter `-> https://github.com/mkonir0/serenity-cucumber-starter this commit demonstrates mentioned issue -> https://github.com/mkonir0/serenity-cucumber-starter/commit/dce05792028f3892761fcb3534b41428f272aa7c I have following feature file ``` Feature: Search by keyword @green Scenario: Searching for 'green' Then assert in double nested step method ``` and following stepdefinition ``` public class SearchStepDefinitions { @Steps private UserSteps userSteps; @Then("assert in double nested step method") public void assertResponseIsCustomHttpsBlockPage() { userSteps.assertInDoubleNestedStepMethod(); } } ``` and following steps ``` public class UserSteps { @Step public void assertInDoubleNestedStepMethod() { assertInStepMethod(); } @Step public void assertInStepMethod() { assertThat(false).isTrue(); } } ``` and following CucumberTestSuite class ``` @Suite @IncludeEngines("cucumber") @SelectClasspathResource("/features") @ConfigurationParameter(key = PLUGIN_PROPERTY_NAME, value = "pretty,timeline:build/test-results/timeline") public class CucumberTestSuite { } ``` when I execute `mvn verify` goal without using the `io.cucumber.core.plugin.SerenityReporterParallel` plugin the tests falsly end up as success ``` [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running starter.CucumberTestSuite @green Scenario: Searching for 'green' # features/search/search_by_keyword.feature:4 10:40:20.563 [ForkJoinPool-2-worker-1] INFO n.thucydides.core.steps.StepEventBus - Loading JUnit Platform configuration parameters from classpath resource [junit-platform.properties]. Then assert in double nested step method # starter.stepdefinitions.SearchStepDefinitions.assertResponseIsCustomHttpsBlockPage() [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.582 s -- in starter.CucumberTestSuite ``` when I execute `mvn verify` goal with using the `io.cucumber.core.plugin.SerenityReporterParallel` plugin the tests end up as failure ``` Searching for 'green'(search-by-keyword;searching-for--green-) -------------------------------------------------------------------------------- [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.174 s <<< FAILURE! -- in starter.CucumberTestSuite [ERROR] Search by keyword.Searching for 'green' -- Time elapsed: 1.122 s <<< FAILURE! org.opentest4j.AssertionFailedError: Expecting value to be true but was false at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) at starter.steps.UserSteps.assertInStepMethod(UserSteps.java:17) at starter.steps.UserSteps$ByteBuddy$caQJP6uq.assertInStepMethod$accessor$01Qi3QF7(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at net.thucydides.core.steps.StepInterceptor.invokeMethod(StepInterceptor.java:519) at net.thucydides.core.steps.StepInterceptor.executeTestStepMethod(StepInterceptor.java:499) at net.thucydides.core.steps.StepInterceptor.runTestStep(StepInterceptor.java:471) at net.thucydides.core.steps.StepInterceptor.runOrSkipMethod(StepInterceptor.java:213) at net.thucydides.core.steps.StepInterceptor.testStepResult(StepInterceptor.java:200) at net.thucydides.core.steps.StepInterceptor.intercept(StepInterceptor.java:92) at net.thucydides.core.steps.ProxyConfiguration$InterceptorDispatcher.intercept(ProxyConfiguration.java:51) at starter.steps.UserSteps$ByteBuddy$caQJP6uq.assertInStepMethod(Unknown Source) at starter.steps.UserSteps.assertInDoubleNestedStepMethod(UserSteps.java:12) at starter.steps.UserSteps$ByteBuddy$caQJP6uq.assertInDoubleNestedStepMethod$accessor$01Qi3QF7(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at net.thucydides.core.steps.StepInterceptor.invokeMethod(StepInterceptor.java:519) at net.thucydides.core.steps.StepInterceptor.executeTestStepMethod(StepInterceptor.java:499) at net.thucydides.core.steps.StepInterceptor.runTestStep(StepInterceptor.java:471) at net.thucydides.core.steps.StepInterceptor.runOrSkipMethod(StepInterceptor.java:213) at net.thucydides.core.steps.StepInterceptor.testStepResult(StepInterceptor.java:200) at net.thucydides.core.steps.StepInterceptor.intercept(StepInterceptor.java:92) at net.thucydides.core.steps.ProxyConfiguration$InterceptorDispatcher.intercept(ProxyConfiguration.java:51) at starter.steps.UserSteps$ByteBuddy$caQJP6uq.assertInDoubleNestedStepMethod(Unknown Source) at starter.stepdefinitions.SearchStepDefinitions.assertResponseIsCustomHttpsBlockPage(SearchStepDefinitions.java:21) at ✽.assert in double nested step method(classpath:features/search/search_by_keyword.feature:5) [INFO] [INFO] Results: [INFO] [ERROR] Failures: [ERROR] Expecting value to be true but was false [INFO] [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0 ``` ### How can we make it happen? Work on this myself and propose a PR (with Serenity BDD team guidance)

[wakaleo]

You need to use this plugin for Serenity 4 and Cucumber 7 regardless of whether you are running the tests in parallel or not.

[mkonir0]

@wakaleo Thanks for help! I didn't notice it in any of the migration guides.

Also in one of our projects, where we're still using junit4 we have a single cucumber runner (which is beiing executed in runtime) -

import org.junit.runner.RunWith;
import io.cucumber.junit.CucumberOptions;
import net.serenitybdd.cucumber.CucumberWithSerenity;

@RunWith(CucumberWithSerenity.class)
@CucumberOptions(
        features = { ... },
        glue = { ... },
        plugin = { "com.oracle.runtime.cucumber.CustomCucumberListener", "io.cucumber.core.plugin.SerenityReporterParallel" })
public class CustomCucumberRunner {
}

when we upgraded to serenity4 and started using plugin io.cucumber.core.plugin.SerenityReporterParallel the tests started to run twice(that wasn't the case in serenity3)

there's seems to be no issue with the glue or the features values.

Could you advise me which of these annotations to use with serenity4 🙏 either this

import net.serenitybdd.cucumber.CucumberWithSerenity;
@RunWith(CucumberWithSerenity.class)

or this one

import io.cucumber.junit.Cucumber;
@RunWith(Cucumber.class)

[wakaleo]

If you are running in JUnit 4 you need to use CucumberWithSerenity with the JUnit 5 legacy mode.