Failed serenity test isn't reported as 'FAILED' in ReportPortal

[divannn]

HI, I use serenity-reportportal-integration library for sending results of test execution to ReportPortal. I noticed strange behaviour. Run the following trivial test case :

@Slf4j @ExtendWith(SerenityJUnit5Extension.class) class TestXXX {

@Test
void test1() {
    log.info(">run 1");
    throw new RuntimeException("fail");
}

@Test
void test2() {
    log.info(">run 2");
    assertThat(1).isEqualTo(2);
}

}

Both tests are failed correctly. But when I see results in ReportPortal - they are both marked as PASSED.

Is this known issue or behavior?

I suspect that com.github.invictum.reportportal.ReportPortalListener doesn't handle failed test properly.

Thank you in advance, -Ivan

[hunteryin]

I found out it only report failed test case if the error/exception happened inside step. If it failed from test directly, the report portal show it as pass. Do we have any any plan to fix this issue?

[grey-rain]

@hunteryin @divannn I know, it's been a while, but I found where it comes from. It is Serenity bug coming from SerenityTestExecutionListener.java class. It seems like Serenity also does some post-processing AFTER listeners invocations to updates results, but no listeners has been invoked to receive updates, that's why you see it as "Failed" in ReportPortal. Long story short, I will try to propose PR to Serenity once I have time. Maybe integration rework will help to fix it as well, but I am not sure in it. In any case, it is not really related to integration implementation.

[grey-rain]

@hunteryin @divannn here's related bug I reported and fixed in Serenity: https://github.com/serenity-bdd/serenity-core/issues/3298 It should resolve the issue in upcoming Serenity and integration versions. Thanks for bringing it up! 👍🏻

[grey-rain]

Reopening until new Serenity version is released and wrapped into integration.

[grey-rain]

Should be resolved with integration version 1.6.1. cc @hunteryin @divannn