Clarify JUnit5 @ParameterizedTest

[fabianlinz]

Using JUnit5 @ParameterizedTest (https://junit.org/junit5/docs/current/user-guide/#writing-tests-parameterized-tests) already "works" (see file below).

Goal

• clarify and document relation to Serenity SerenityParameterizedRunner (http://thucydides.info/docs/serenity-staging/#_data_driven_tests)

[qand90]

Hello @fabianlinz , I am facing one problem with @ParameterizedTest. Tests are marked as Successful even though the assertion failed. An issue occurred when assertion called inside @Step method from @Steps class . This is not reproducible with @Test only with @ParameterizedTest. Also not reproducible if assertion called in Test class itself.

public class Test extends BaseTest { @ParameterizedTest @ValueSource(ints= {1,2}) public void test(int i) { pageSteps.assert(); } }`

---

@Disabled('Base test is abstract') @SerenityTest public abstract class BaseTest {

@Steps protected PageSteps pageSteps; }

---

public class PageSteps extends ScenarioSteps { @Step public void asert(){ Assertions.assertTrue(false); } }

[qand90]

To FIx this issue with parametrized test you need to implement InvocationInterceptor.interceptTestTemplateMethod() in https://github.com/fabianlinz/serenity-junit5/blob/master/serenity-junit5/src/main/java/net/serenitybdd/junit5/extension/SerenityStepExtension.java

See https://github.com/junit-team/junit5/issues/2338

Currently, I have implemented my own extension with this method and it works.

[fabianlinz]

@qand90 thanks for the feedback and the example!

You are right, if a parametrised test fails inside a @Step method, the test is passing from a JUnit perspective while the Serenity report shows the test as failing. With the fix you suggested (net.serenitybdd.junit5.extension.SerenityStepExtension#interceptTestTemplateMethod) the integration with the Serenity step handling works properly. I will try to find some time in the next days to back this by some tests.

If I remember correctly two other things to be aware of are:

1. for JUnit4 a method in a parameterised Serenity test (@RunWith(SerenityParameterizedRunner.class) is reported as one scenario having an example table, while for JUnit5 each test method invocation is currently reported as an individual scenario.
2. the default name for a parametrised test does not include the test method (org.junit.jupiter.params.ParameterizedTest#DEFAULT_DISPLAY_NAME). If the test class has multiple methods the context get's lost. Currently this can be addressed by defining a custom name (e.g. @ParameterizedTest(name = ParameterizedTest.DISPLAY_NAME_PLACEHOLDER + ":" + ParameterizedTest.DEFAULT_DISPLAY_NAME))

[mukesh90hz]

@qand90 - Even I am getting the same issue. Tests are getting passed even if the test failed using @ParameterizedTest I have implemented my own extension but still, Tests are getting passed.

import net.thucydides.core.steps.BaseStepListener; import net.thucydides.core.steps.StepEventBus; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.extension.*; import org.opentest4j.TestAbortedException;

import java.lang.reflect.Method;

import static net.thucydides.core.steps.StepAnnotations.injector; import static net.thucydides.core.steps.StepEventBus.getEventBus; import static net.thucydides.core.steps.StepFactory.getFactory;

public class SerenityStepInterceptor implements Extension, InvocationInterceptor, BeforeEachCallback {

@Override
public void beforeEach(final ExtensionContext context) {
    injector().injectScenarioStepsInto(context.getRequiredTestInstance(), getFactory());
}

@Override
public void interceptTestMethod(final Invocation<Void> invocation,
                                final ReflectiveInvocationContext<Method> invocationContext,
                                final ExtensionContext extensionContext) throws Throwable {
    try {
        invocation.proceed();
    } catch (final Throwable assertionError) {

        if (noStepInTheCurrentTestHasFailed()) {
            Assertions.fail();
        }
    }

    throwStepFailures(extensionContext);
    throwStepAssumptionViolations(extensionContext);
}

private boolean noStepInTheCurrentTestHasFailed() {
    return !getEventBus().aStepInTheCurrentTestHasFailed();
}

private void throwStepFailures(final ExtensionContext extensionContext) throws Throwable {
    final BaseStepListener baseStepListener = baseStepListener();
    if (baseStepListener.aStepHasFailed()) {
        throw baseStepListener.getTestFailureCause().toException();
    }
}

private void throwStepAssumptionViolations(final ExtensionContext extensionContext) {
    final StepEventBus eventBus = getEventBus();
    if (eventBus.assumptionViolated()) {
        throw new TestAbortedException(eventBus.getAssumptionViolatedMessage());
    }
}

private BaseStepListener baseStepListener() {
    return getEventBus().getBaseStepListener();
}

}

Can you please send me the custom implementation?

[mukesh90hz]

@fabianlinz @wakaleo @qand90 - I am using serenity bdd with junit5.

I am calling custom Interceptor like this

@SerenityTest @TestMethodOrder(MethodOrderer.OrderAnnotation.class) @ExtendWith(SerenityStepInterceptor.class) public class PeopleTest extends BaseConfig{}

@Tag("Regression") @ParameterizedTest @ValueSource(strings = {"email"})

[fabianlinz]

Hi @mukesh90hz,

this project does not support Junit5 Parameterised tests properly. Last year I started to look into the comments from @qand90 above and changed some things but unfortunately got distracted from this issue (sorry!). Anyway proper support would be to report the test as one scenario with an example table entry for each run (as Serenity JUnit4 does it @RunWith(SerenityParameterizedRunner.class).

Now that time moved on I don't think it makes much sense to spend effort on improving this in this project, as Serenity now includes Junit5 support out of the box. As far as I know parameterised tests are also supported (and use example tables as mentioned before). So I would suggest that you give that one a try (https://github.com/serenity-bdd/serenity-core/tree/master/serenity-junit5).

[scormaq]

Don't really want to necropost here, but since serenity-junit5 still shows failed @ParameterizedTest as passed, and internet search shows this page, I would like to share my workaround to address this.

I use this extension to fail parameterized tests manually:

import net.thucydides.core.steps.StepEventBus;
import net.thucydides.model.domain.TestResult;
import org.junit.jupiter.api.extension.AfterEachCallback;
import org.junit.jupiter.api.extension.ExtensionContext;

    public class ParameterizedSerenityTestExtension implements AfterEachCallback {

        @Override
        public void afterEach(ExtensionContext context) {
        if (context.getTestMethod().isPresent()) {

            // workaround to fix serenity false-positive parameterized tests
            boolean isTestFailed = !StepEventBus.getEventBus().resultSoFar().get().equals(TestResult.SUCCESS);
            boolean isErrorPresent = StepEventBus.getEventBus().getBaseStepListener().getTestFailureCause() != null;
            boolean isTestParameterized = context.getTestMethod().get().getParameters().length > 0;
            if (isTestFailed && isErrorPresent && isTestParameterized) {
                throw StepEventBus.getEventBus().getBaseStepListener().getTestFailureCause()
                        .asRuntimeException();
            }
        }
    }
}