ATF Test Run Task requires browser/OS fields to be filled in to successfully run if UI Test

[chiarng]

From Ramya: ATF Issue: The ATF task in the pipeline was expecting all the browser and OS parameters to be filled in, unlike its Flow Designer equivalent which would run successfully just with the Test Suite sys_id and Instance Name. Until I filled in all the details, it kept skipping the ATF Tests and the task failed. It would be helpful if browser and OS parameter fields are made MANDATORY in the Azure task.

[ramyaramdasan]

Hi Chiarng,

Thanks for posting it. A small correction that the browser and OS parameter fields should be made mandatory, not optional. Sorry about the mistake.

Thanks Ramya Ramadasan

[chiarng]

Sorry, @ramyaramdasan, that was my fault! I had made an incorrect assumption in my understanding from your first report. Thanks for clarifying on call today! For reference to anyone else reading this in the future - Neither the API nor the Spoke require the browser/OS/etc. fields to be filled in to run ATF Test Suites, only sys_id is required. However, it sounds like there are some types of tests that won't run unless you do fill it in.

Next Steps:

• Chiarng to clarify with ATF team on which tests require browser/OS input from API to successfully run
• Chiarng to think about how to inform developer users that if their ATF Test Suite includes these tests, then they must fill in these currently optional parameters, because they are then required.

[chiarng]

I note the API documentation for these parameters says "This value must match what is specified in the scheduled client test runner." https://developer.servicenow.com/dev.do#!/reference/api/orlando/rest/cicd-api#cicd-POST-testsuite-run

[chiarng]

So I did a bit more testing for the ATF team, just documenting my results here. I set up a new Test Suite with the OOTB "Basic UI Test". I had a Client Test Runner opened up in Chrome browser on MacOS. I used the REST API Explorer to fire off a POST https://chiarngtestdemo.service-now.com/api/sn_cicd/testsuite/run with only the test_suite_sys_id parameter filled in.

Response was 200.

However using the progress api endpoint to pull that run gets the following response

{
  "result": {
    "links": {
      "progress": {
        "id": "1580934bdb18a050874fccd8139619af",
        "url": "https://chiarngtestdemo.service-now.com/api/sn_cicd/progress/1580934bdb18a050874fccd8139619af"
      },
      "results": {
        "id": "d580934bdb18a050874fccd813961976",
        "url": "https://chiarngtestdemo.service-now.com/api/sn_cicd/testsuite/results/d580934bdb18a050874fccd813961976"
      }
    },
    "status": "4",
    "status_label": "Cancelled",
    "status_message": "All suites were canceled",
    "status_detail": "A scheduled client test runner that satisfies these client constraints is not available: | Browser name: Any | Browser version: any | OS Name: Any | OS Version: any",
    "error": "",
    "percent_complete": 100
  }
}

So there's something going on there.

[chiarng]

Sigh. Ok apparently a "scheduled client test runner" is different from a regular "client test runner". Have to have a browser window pointed to https://chiarngtestdemo.service-now.com/atf_test_runner.do?sysparm_scheduled_tests_only=true&sysparm_nostack=true

Back to debugging...

[chiarng]

So using the API explorer DOES work when I have a "scheduled client test runner" running.. and passing in just the test suite sys_id (containing ui tests). I did discover that the progress api endpoint will return the following until the test suite is finished running though.

{
  "result": {
    "status": "3",
    "status_label": "Failed",
    "status_message": "Missing CI/CD Role (sn_cicd.sys_ci_automation)",
    "status_detail": "",
    "error": ""
  }
}

At which point, after the test suite is complete in the scheduled client test runner, it'll return the more expected

{
  "result": {
    "links": {
      "progress": {
        "id": "52b173c3dbd8a050874fccd813961922",
        "url": "https://chiarngtestdemo.service-now.com/api/sn_cicd/progress/52b173c3dbd8a050874fccd813961922"
      },
      "results": {
        "id": "d6b1f707dbd8a050874fccd81396193c",
        "url": "https://chiarngtestdemo.service-now.com/api/sn_cicd/testsuite/results/d6b1f707dbd8a050874fccd81396193c"
      }
    },
    "status": "2",
    "status_label": "Successful",
    "status_message": "Suite passed",
    "status_detail": "Suite passed",
    "error": "",
    "percent_complete": 100
  }
}

[chiarng]

After much digging. https://github.com/ServiceNow/servicenow-cicd-azure-extension/blob/082c1a407fa5df1be2753a2206dc44e52d322bb4/src/extension/Tasks/TestRun/task.json#L53

By defaulting the browser_name parameter in the API calls to value "Any" when nothing other than test_suite_sys_id is submitted where the Test Suite has UI Tests, the progress API will return "A scheduled client test runner that satisfies these client constraints is not available: | Browser name: Any | Browser version: any | OS Name: Any | OS Version: any" because there's no "Any" scheduled client test runners.

[chiarng]

But in our own documentation https://developer.servicenow.com/dev.do#!/reference/api/orlando/rest/cicd-api#cicd-POST-testsuite-run

[chiarng]

In conclusion, either the documentation is wrong, because the value "Any" cannot be supplied to browser_name by itself when the only other parameter provided is test_suite_sys_id for an ATF Test Suite that includes a UI Test dependent on a Scheduled Client Test Runner, OR the documentation is right and the feature should accept browser_name with value "Any" as the only other parameter.

[chiarng]

For example https://chiarngtestdemo.service-now.com/api/sn_cicd/testsuite/run?test_suite_sys_id=298f8fc7db18a050874fccd813961918&browser_name=Any https://chiarngtestdemo.service-now.com/api/sn_cicd/progress/536591db1bd860506a6855ff5b4bcb19

{
    "result": {
        "links": {
            "progress": {
                "id": "536591db1bd860506a6855ff5b4bcb19",
                "url": "https://chiarngtestdemo.service-now.com/api/sn_cicd/progress/536591db1bd860506a6855ff5b4bcb19"
            },
            "results": {
                "id": "5f659d1f1bd860506a6855ff5b4bcb04",
                "url": "https://chiarngtestdemo.service-now.com/api/sn_cicd/testsuite/results/5f659d1f1bd860506a6855ff5b4bcb04"
            }
        },
        "status": "4",
        "status_label": "Cancelled",
        "status_message": "All suites were canceled",
        "status_detail": "A scheduled client test runner that satisfies these client constraints is not available: | Browser name: Any | Browser version: any | OS Name: Any | OS Version: any",
        "error": "",
        "percent_complete": 100
    }
}

[ramyaramdasan]

Thanks Chiarng! I have modified my pipeline with this fix and it works :) No more Chrome version updates...yay!