Test case is still considered successful in the pipeline of Jenkins even by failed steps

[ValeriusSchmidt]

First encounter: Testcase 1386

Method that causes this behaviour in Jenkins: Under notifications, where a list of all pop-up messages is stored

Steps to reproduce

1. Write a test with @Test annotation that includes expectNotificationTextToContainList("Enter your message(s) here") //Be aware that you need a test which triggeres multiple pop-up windows in a short timeframe
2. Execute the test using: mvn test
3. Observe BUILD SUCCESS instead of BUILD FAILURE even though the log contains Test failure or Step failure

[OliverHoenig]

After a one-to-one session with Valerius, the issue could be abstracted as follows:

Within the test case, assertTrue (from QTAF) was called in one step. This step of the test case fails. The following anomalies were found:

• During local execution, the logs were displayed correctly in the console. The step failed correctly and the test case is also logged as faild.

• During execution in the pipeline, the step was correctly logged as fail. However, it was not displayed that the entire test case failed. (But it was also not logged that the test case was successfull.) In Jenkins, the test is displayed as supposedly correct, although it is actually failed. The upload to Xray works as desired. (Both the step and the test case are displayed as failed) Even deactivating the Xray upload in the pipeline did not result in the local behaviour being reported. It was noticeable that an error occurred during execution in the pipeline, which suggests that a scenarioLog object, which is null, is accessed when the scrennshot path is set. It is currently unclear in which circumstances it can happen that a scenarioLog is accessed which is null. This error did not occurred during the local execution.

It is also not clear whether this is a QTAF bug or whether configurations in the pipeline cause side effects that lead to this behaviour.

The behaviour is now being analysed.

[OliverHoenig]

I have tried to generalise the scenario in which the suspected bug might occur. Here is the class which contains steps:

package de.qytera.tests.issue311;

import de.qytera.qtaf.core.guice.annotations.Step;
import de.qytera.qtaf.testng.context.QtafTestNGContext;

public class DummySteps extends QtafTestNGContext {
    @Step(
            name="dummy Step",
            description = "dummy step for mocking the behavior of issue 311"
    )
    public void trueDummyStep() {
        assertTrue(true, "Step should be true");
    }
    public void falseDummyStep() {
        assertTrue(false, "Step should be true");
    }
}

And a here the used test class:

package de.qytera.tests.issue311;

import de.qytera.qtaf.core.config.annotations.TestFeature;
import de.qytera.qtaf.testng.context.QtafTestNGContext;
import org.testng.annotations.Test;

@TestFeature(
        name = "TestCase One",
        description = "First test"
)
public class Issue311Test extends QtafTestNGContext {

    @Test(testName  = "simplifiedBugScenario", description = "most abstracted scenario the bug could potential occur")
    public void simplifiedBugScenario(){
        DummySteps dummySteps = load(DummySteps.class);
        dummySteps.falseDummyStep();
    }
}

I could not reproduce the bug locally on my device. However, the execution of the above testcase (simplifiedBugScenario()) resulted in a NullPointerException when the test was executed in your project. Remarkably, only if dummySteps.falseDummyStep() was called and not if dummySteps.trueDummyStep() gets called instead in the test case.

There seems to be a NullPointerException in your HtmlDumpSubscriber, which implements IEvenetSubscriber. Inside the method getHtmlDumpDestitnationPath() the following line leads to the following exception:

String name = stepExecutionInfo.getMethodInvocation().getMethod().getName();

java.lang.NullPointerException: Cannot invoke "org.aopalliance.intercept.MethodInvocation.getMethod()" because the return value of "de.qytera.qtaf.core.guice.invokation.StepExecutionInfo.getMethodInvocation()" is null

[ValeriusSchmidt]

To reproduce this behaviour in the pipeline I created a new branch "AssertTestBranch" (without quotes). The test target of reproduction is testcase 1386.

[qytera-development]

The solution is to check whether a class is annotated with the specific listener in the beforeInvocation method:

    public void beforeInvocation(IInvokedMethod invokedMethod, ITestResult result) {
        LocalTime currentTimeUTC = LocalDateTime.now()
                .atZone(ZoneId.systemDefault())
                .withZoneSameInstant(ZoneId.of("UTC+2"))
                .toLocalTime();
        Annotation[] anns = invokedMethod.getTestMethod().getRealClass().getAnnotations();

        // Check if test class has Listener annotations
        for (Annotation ann : anns) {
            if (ann instanceof Listeners) {
                Class<? extends ITestNGListener>[] listeners = ((Listeners) ann).value();
                for (Class<? extends ITestNGListener> listener : listeners) {
                    // Check if one of the Listeners is an instance of BindingClassListener
                    if (listener.equals(BindingGasListener.class)) {
                        if (currentTimeUTC.isBefore(START_TIME)) {
                            throw new SkipException("Binding offer gas tests must be started after %s".formatted(START_TIME.toString()));
                        }
                    }
                }
            }
        }

    }

[OliverHoenig]

@qytera-development Are changes in the QTAF-repo necessary, or did I understand correctly that the changes in your implementation are sufficient? Have the changes already been made, and has this resolved the issue? If yes: Can this issue be closed?