Serenity 3.5.0 - junit5 - @Manual tests are executed but they should not

[DevQuandt]

I noticed while upgrading from junit4 to Serenity junit5 that with the junit5 Extension the @Manual tests are behaving differently.

Example:

Junit4

    @Test
    @Manual(result = SUCCESS)
    public void manual_test() {
       throw new UnsupportedOperationException();
    }

Result: Test passed -> Code was not executed -> no Exception is thrown (report shows success)

Junit5

    @Test
    @Manual(result = SUCCESS)
    void manual_test() {
        throw new UnsupportedOperationException();
    }

Result: Test failed -> Code was executed -> Exception is thrown (nevertheless the serenity report shows success)

I would guess that the junit5 extension should also not execute the code when the Test is annotated with @Manual

[wakaleo]

I don't think Manual tests are supported yet in JUnit 5

[globalworming]

by coincidence i stumbled upon https://qaautomation.expert/2021/10/05/manual-tests-in-serenity-with-junit5/ yesterday

executing manual annotated scripts is actually great IMHO, because it allows you to document steps and substeps if you like

[DevQuandt]

@globalworming I agree for document steps the tests should be executed but when marked as "result = SUCCESS" it should result in a green test. (Ignoring all Exceptions)

@wakaleo Is there a plan to support this in the near future?

[DevQuandt]

@globalworming The report shows successful as in the screenshots but my problem is that my (gradle) build is failing because there are test failures.

Running the examples above with junit4 and junit5 results in

Junit5: Process finished with exit code 255 -> Test failed Junit4: Process finished with exit code 0 -> Test ignored/success

[globalworming]

why do you get exceptions? i think you don't really want to execute something that really does something (like throwing exceptions)

or have you marked them as "failing manual test" and the build fails? you can force gradle to continue building even when there are test failures, or when you call gradle test || true

[DevQuandt]

@globalworming I have normal tests using steps but also manual tests which are using some of these steps, too. In my current case the step is expecting a Serenity session variable which is not set, which results in a NullpointerException. (Because given+then is not implemented but when is used from an implemented step)

Example:

    @Test
    void example_1() {
        o.given_a_new_user(); //implemented

        o.when_opening_the_page(); //implemented

        o.then_the_page_is_opened(); //implemented
    }

and

    @Test
    @Manual(result = SUCCESS)
    void example_2() {
        o.given_an_already_created_user(); //not implemented

        o.when_opening_the_page(); //implemented

        o.then_the_other_page_is_opened(); //not implemented
    }

I want the build to fail, when there are actually (serenity) tests which are failing.

[globalworming]

workaround.. does it help when you in the manual tests use a specific actor Actor actor = new MyActor("tester") and in the MyActor extend the Actor class and override the attemptsTo method and others to not execute any task you pass in? that won't report any steps though...

[DevQuandt]

@globalworming I guess there will be some ways how I can implement a (ugly) workaround. But of course my prefered way would be that the junit5 extension support this kind of behaviour. But thanks for your suggestions!

[zzoubian]

Perhaps, you can use assertThrows().

Assertions.assertThrows(NullPointerException.class, () -> { methodThatThrowsNullPointer(); });

[AlexRiaboshapka]

Or you can say that the NullPointer Exception is expected in the test using this annotation @Test(expected = NullPointerException.class)