Misleading/Wrong Serenity report in case of exception during hook execution

[datentyp]

Hello,

Serenity reports buggy scenario results in case an exception occurs during the execution of some (@Before) hook. Some scenarios are reported as ignored, some as passed.

Imagine an exception gets thrown in a before hook:

    @Before
    public void setup() {
        System.out.println("Executing @Before Hook");
        // simulate some external operation that fails for some reason, e.g. database setup/cleanup
        throw new RuntimeException("Oops! Some exception happened during hook execution!");
    }

Then Cucumber correctly reports:

3 Scenarios (3 failed)
12 Steps (12 skipped)
0m0.002s

But Serenity instead comes up with this results:

2 test scenarios (3 tests in all, including 2 rows of test data)
| 1 passed | 1 ignored | 

A sample project to reproduce the problem can be found at: https://github.com/datentyp/bugs-cucumber-hooks-and-serenity-report

The sample project does contain more details on the bug and the expected behavior. I've included the generated report for a quick view as well.

Regards, Wolfgang

[datentyp]

Hi,

Any feedback on this issue from your side?

Note that I've updated serenity (1.2.5-rc.2) and serenity-cucumber (1.1.28) to their latest versions to demonstrate that this issue still exists.

Thanks,

[wakaleo]

If it's not marked as fixed, then most likely no one has picked it up yet.

[wakaleo]

Fixed in the next release (1.6.14). Note that errors in hook logic occur before the steps are executed, so the error will still happen even for scenarios marked with @ignore or @pending. However in these cases the tests will be reported as Ignored or pending.

[datentyp]

Hi John,

Thx! I've verified the fix by updating my sample project at https://github.com/datentyp/bugs-cucumber-hooks-and-serenity-report

Looks really good :-)

[wakaleo]

Thanks for the update :-)