JUnit not reporting correct numbers

[Mark-J-Lawrence]

Describe the bug

I'm doing a local run of a test class, containing two simple test cases, one which I'm forcing a fail via a particular assert. I was assessing what comes out in the json & JUnit reports as I plan to write a simple 'Galasa report - to - markdown' tool (which will likely be used in the CICS-TS test bot in the future), however the failure only appears in the Json, not the JUnit, which was the preferable one to use.

Steps to reproduce

1. I'm using the following command to run some local tests galasactl runs submit local --log - --obr mvn:com.ibm.cics.sdv/com.ibm.cics.sdv.tests.obr/0.0.1/obr --class com.ibm.cics.sdv.tests/com.ibm.cics.sdv.tests.featureA.TestClassOne --remoteMaven https://development.galasa.dev/main/maven-repo/obr --overridefile ./overrides.properties --reportjunit ./junitReport.xml --reportjson ./jsonreport.json
2. The run ends with

   
   2024/02/20 14:16:21 Run L517 has finished(Failed) - /com.ibm.cics.sdv.tests/com.ibm.cics.sdv.tests.featureA.TestClassOne
   submitted-time(UTC) name requestor    status  result test-name
   2024-02-20 14:13:34 L517 marklawrence running Failed /com.ibm.cics.sdv.tests/com.ibm.cics.sdv.tests.featureA.TestClassOne

Total:1 Failed:1 2024/02/20 14:16:21 Json test report written to ./jsonreport.json 2024/02/20 14:16:21 Junit XML test report written to ./junitReport.xml Error: GAL1017E: Not all runs passed. 1 failed. 2024/02/20 14:16:21 GAL1017E: Not all runs passed. 1 failed. 2024/02/20 14:16:21 Exit code is 2

## Expected behavior
The reports show that `1` test class was ran, with `1` failure.

## Actual behavior

The Json report contains:

{ "tests": [ { "name": "L517", "bundle": "com.ibm.cics.sdv.tests", "class": "com.ibm.cics.sdv.tests.featureA.TestClassOne", "stream": "", "obr": "", "status": "running", "queued": "2024-02-20 14:13:34.23717 +0000 UTC", "requestor": "marklawrence", "result": "Failed", "overrides": { "cicsts.dse.tag.SDVCICS.applid": "IYK2ZPD1", "cicsts.dse.tag.SDVCICS.version": "750", "cicsts.provision.type": "DSE", "sdv.cicsTag.SDVCICS.hlq": "ANTZ.CICS.TS.DEV.INTEGRAT", "sdv.cicsTag.SDVCICS.port": "20292", "sdv.recording.enable": "true", "sdv.zosImage.MV2C.credentials": "SDVUSER1,SDVUSER2", "sdv.zosImage.MV2C.credentials.SDVUSER1.role": "TELLER", "sdv.zosImage.MV2C.credentials.SDVUSER2.role": "ADMIN", "sdv.zosImage.MV2C.credentials.SDVUSER3.role": "OPERATOR" }, "tests": [] } ] }

The JUnit report contains:

<?xml version="1.0" encoding="UTF-8" ?>

According to the JUnit report, there were `0` tests in the test class, and `0` failures, which is not true.

[techcobweb]

Thanks for raising this. We will look at it shortly.

[Akyiaa]

The JUnit report only seems to show the results of the TestRun tests. For example if there is another run with 2 tests/testcases:

{
    "tests": [
    {
      "name": "myTestRun",
      "bundle": "myBundle",
      "class": "com.myco.MyClass",
      "stream": "myStream",
      "obr": "",
      "status": "myStatus",
      "queued": "",
      "requestor": "",
      "result": "FAILED",
      "overrides": {},
      "tests": [
                {
                    "method": "method1",
                    "result": "passed",
                },
                {
                    "method": "method2",
                    "result": "failed",
                },
            ]
    }
    ]
}

Then the following would be printed out:

<?xml version="1.0" encoding="UTF-8" ?>
<testsuites id="myGroup" name="Galasa test run" tests="2" failures="1" time="0">
    <testsuite id="myTestRun" name="myStream/myBundle/com.myco.MyClass" tests="2" failures="1" time="0">
        <testcase id="method1" name="method1" time="0"></testcase>
        <testcase id="method2" name="method2" time="0">
            <failure message="Failure messages are unavailable at this time" type="Unknown"></failure>
        </testcase>
    </testsuite>
</testsuites>

Similarly, if there were 2 runs with the first having 2 passed tests and the second having 1 passed and 1 failed test:

{
    "tests": [
    {
      "name": "myTestRun1",
      "bundle": "myBundle1",
      "class": "com.myco.MyClass1",
      "stream": "myStream1",
      "obr": "",
      "status": "myStatus1",
      "queued": "",
      "requestor": "",
      "result": "PASSED",
      "overrides": {},
      "tests": [
                {
                    "method": "method1.1",
                    "result": "passed",
                },
                {
                    "method": "method1.2",
                    "result": "passed",
                },
            ]
    },
        {
      "name": "myTestRun2",
      "bundle": "myBundle2",
      "class": "com.myco.MyClass2",
      "stream": "myStream2",
      "obr": "",
      "status": "myStatus2",
      "queued": "",
      "requestor": "",
      "result": "FAILED",
      "overrides": {},
      "tests": [
                {
                    "method": "method2.1",
                    "result": "passed",
                },
                {
                    "method": "method2.2",
                    "result": "failed",
                },
            ]
    },
    ]
}

The result would be:

<?xml version="1.0" encoding="UTF-8" ?>
<testsuites id="myGroup" name="Galasa test run" tests="4" failures="1" time="0">
    <testsuite id="myTestRun1" name="myStream1/myBundle1/com.myco.MyClass1" tests="2" failures="0" time="0">
        <testcase id="method1.1" name="method1.1" time="0"></testcase>
        <testcase id="method1.2" name="method1.2" time="0"></testcase>
    </testsuite>
    <testsuite id="myTestRun2" name="myStream2/myBundle2/com.myco.MyClass2" tests="2" failures="1" time="0">
        <testcase id="method2.1" name="method2.1" time="0"></testcase>
        <testcase id="method2.2" name="method2.2" time="0">
                    <failure message="Failure messages are unavailable at this time" type="Unknown"></failure>
                </testcase>
    </testsuite>
</testsuites>

---

The conclusion is that the output from the command Mark supplied is actually correct. The JUnit report is saying there are no tests for that run, hence no failures. One possible improvement from this finding is to make the JUnit report if the run itself (TestSuite) failed alongside if its testcases failed.

[techcobweb]

If there were no tests for that run, how come the run failed ? @Mark-J-Lawrence can you pls re-run and show us the log associated with the test run ?

[Mark-J-Lawrence]

Here is a run I've just done, using development 0.32.0. I've forced the two test classes to fail based on an assertion.

console.log

test-report.json

test-report.xml.zip

[Mark-J-Lawrence]

This is the test code, if it helps:

package com.ibm.cics.sdv.tests.featureA;

import static org.assertj.core.api.Assertions.*;
import dev.galasa.core.manager.*;
import dev.galasa.sdv.manager.SdvUsers;
import dev.galasa.sdv.manager.SdvUser;
import dev.galasa.cicsts.*;
import dev.galasa.Test;

@SdvUsers({
    @SdvUser(credentialsTag="SDVUSER1", role="TESTROLE"),
    @SdvUser(credentialsTag="SDVUSER2"),
    @SdvUser(credentialsTag="SDVUSER3"),
})

@Test
public class TestClassOne {

    @CicsRegion(cicsTag = "SDVCICS")
    public ICicsRegion myRegion;

    @CicsTerminal(cicsTag = "SDVCICS", loginCredentialsTag = "SDVUSER1")
    public ICicsTerminal cicsTerminalA;

    @CicsTerminal(cicsTag = "SDVCICS", loginCredentialsTag = "SDVUSER2")
    public ICicsTerminal cicsTerminalB;

    @CicsTerminal(cicsTag = "SDVCICS", loginCredentialsTag = "SDVUSER3")
    public ICicsTerminal cicsTerminalC;

    @CoreManager
    public ICoreManager core;

    @Test
    public void tellerUsesCEDA() throws Exception {

        cicsTerminalA.type("CEDA DI G(SDVGRP)").enter().waitForTextInField("SDVXSDT");

        assertThat(cicsTerminalA.searchText("SDVSDT"))
                .as("Expectation to see SDVXSDT in terminal").isTrue();
        cicsTerminalA.pf3();
    }

    @Test
    public void adminUsesCEDA() throws Exception {

        cicsTerminalB.type("CEDA DI G(SDVGRP)").enter().waitForTextInField("SDVXSDT");

        assertThat(cicsTerminalB.searchText("SDVXSDT"))
                .as("Expectation to see SDVXSDT in terminal").isTrue();
        cicsTerminalB.pf3();
    }

}

[Mark-J-Lawrence]

Update - I've just looked at the JUnit report for the same tests running on an ecosystem, and the reported numbers looked correct. The report looks good to me. So this is just a problem with running tests locally.

[Akyiaa]

Thanks for the updates everyone. I am debugging a sample test locally and although the information is stored correctly in the structure.json file of the run in the RAS, the tests/methods for each run are neither being reported in the json nor junit reports. So I am working on that now.