[Improvement] Overwrite "Generic Test Definition" using data provider param in TestNG

[vlad230]

I am looking into how I could overwrite the "Generic Test Definition" (created by default as: class name + "." + test method name from TestNG xml) in order to have individual tests listed in Jira Xray (not under the Execution Details) such that I can:

• see the correct "Overall Execution Status" (x passed, x failed) right now this is masked (if you have 1 test method with data provider having 100 different inputs, it will say 1 test passed/failed not 100 passed/failed)
• be able to mark individual tests (which are part of the data provider input) as passed/failed/skipped after a manual inspection if needed

I was able to re-write the TestNG attribute like this: test-instance-name="testMethodName_ID1"" where "testMethodName" is the actual @Test method name and "ID1" is a value coming from a data provider. So, I would like to have this "Generic Test Definition" be updated with class name + "." + test-instance-name (when available, otherwise fallback to the current approach) such that these results will be displayed separately.

Any help would be appreciated.

[bitcoder]

From my understanding, you have a data-driven automated test that you would like to see the individual results in Xray. Currently, there's no direct support for it with the existing TestNG XML report. Xray still lacks datasets on "generic"/unstructured tests, so you wont be able to see it right now. One option could be to use "manual" (i.e. step based) Tests, for which Xray supports the concept of datasets; then use the Xray JSON report format to report the results for each data row. Not sure if have understood correctly your request given the last 2 phrases of your request... please feel free to clarify :)

[vlad230]

@bitcoder can you give me more details regarding your suggested solution? I'm not familiar with it.

[bitcoder]

The Xray JSON format is an alternate format that Xray supports. More info here. The previous links are for Xray on Jira cloud. You can use the "iterations" field then to report the results for each data row. At this time I don't have a ready to show example; if I have some time I will add it here later on

[bitcoder]

ar btoa = require('btoa');
var axios = require('axios');
var fs = require('fs');
var FormData = require('form-data');

var xray_cloud_base_url = "https://xray.cloud.getxray.app/api/v2";
var client_id = process.env.CLIENT_ID || "215FFD69FE4644728C72182E00000000";
var client_secret = process.env.CLIENT_SECRET || "1c00f8f22f56a8684d7c18cd6147ce2787d95e4da9f3bfb0af8f02ec00000000";

var authenticate_url = xray_cloud_base_url + "/authenticate";

axios.post(authenticate_url, { "client_id": client_id, "client_secret": client_secret }, {}).then( (response) => {
    console.log('success');
    var auth_token = response.data;
    console.log("AUTH: " + auth_token);

    const report_content = fs.readFileSync("xray_cloud_manual_with_dataset.json").toString();
    console.log(report_content);

    var endpoint_url = xray_cloud_base_url + "/import/execution";
    axios.post(endpoint_url, report_content, {
    headers: { 'Authorization': "Bearer " + auth_token, "Content-Type": "application/json" }
    }).then(function(res) {
    console.log('success');
    console.log(res.data.key);
    }).catch(function(error) {
    console.log('Error submiting results: ' + error);
    });
}).catch( (error) => {
    console.log('Error on Authentication: ' + error);
});

contents of xray_cloud_manual_with_dataset.json file:

{
    "info" : {
        "summary" : "Execution of automated tests",
        "description" : "This execution is automatically created when importing execution results from an external source",
        "project" : "CALC",
        "version" : "v1.0",
        "revision" : "1234",
        "startDate" : "2014-08-30T11:47:35+01:00",
        "finishDate" : "2014-08-30T11:53:00+01:00",
        "testEnvironments": ["Chrome"]
    },
    "tests" : [
        {
            "testKey" : "CALC-2690",
            "start" : "2014-08-30T11:47:35+01:00",
            "finish" : "2014-08-30T11:50:56+01:00",
            "status" : "PASSED",

            "testInfo": {
                "summary": "Strong password validation",
                "type": "Manual",
                "projectKey": "CALC",
                "steps": [
                    {
                        "action": "Open the Change Password screen by selecting option \"My Profile > Password\"",
                        "data": "",
                        "result": ""
                    },
                    {
                        "action": "Fill the password fields with data",
                        "data": "Current Password: ${Password}\nNew Password: ${Password}\nConfirm New Password: ${Password}",
                        "result": "The new password is: ${Valid}\nError:\n${Message}"
                    }
                ]
            },

            "iterations": [
              {
                "name": "Iteration 1",
                "parameters": [
                    {
                        "name": "Password",
                        "value": "2635ftvu23v7t!09"
                    },
                    {
                        "name": "Valid",
                        "value": ""
                    }
                ],
                "log": "Password changed successfully",
                "status": "PASSED",
                "steps": [
                    {
                        "actualResult": "",
                        "status": "PASSED"
                    },
                    {
                        "actualResult": "Password changed successfully",
                        "status": "PASSED"
                    }
                ]
              },
              {
                "name": "Iteration 2",
                "parameters": [
                    {
                        "name": "Password",
                        "value": "123123"
                    },
                    {
                        "name": "Valid",
                        "value": "Not Valid"
                    },
                    {
                        "name": "Message",
                        "value": "Password is too simple."
                    }
                ],
                "log": "Password validation check failed. Password too simple!",
                "status": "FAILED",
                "steps": [
                    {
                        "actualResult": "",
                        "status": "PASSED"
                    },
                    {
                        "actualResult": "Password too simple!",
                        "status": "FAILED"
                    }
                ]
              }
            ]
        }
    ]
}

[vlad230]

@bitcoder Ok, I see what you mean, thanks for the example. The problem is that I have tens of thousands of data points that I execute with TestNG data provider so, it's not feasible to create the Json manually.

The change the Xray team needs to do for this to work is minimal: just change the 'Generic Test Definition' approach on which the Jira 'Test' items are created to use the 'test-instance-name' attribute if available. I've create a support ticket here: https://jira.getxray.app/servicedesk/customer/portal/2/SUPPORT-58540 hopefully they will fix it.

[bitcoder]

I'll close this issue as this request falls under the Xray product itself, and on how it handles the TestNG XML report file during the import process. Nevertheless, on the short term a new enhancement is expected to this project in order to generate a Xray JSON report with more flexibility but that will land on its own issue :)

[bitcoder]

Just to let you know that I've implemented a new reporter (beta), that generates test results in Xray JSON format. The documentation is on the master branch. The Xray JSON reporter won't generate distinct tests for each data row. Instead, it will provision one Test issue with a dataset, and then on the test run you will be able to see the results for each input parameter combination. I invite you to give it a try