Allow Cross Browser Testing in Continuous Integration Environment

[ejayaraman]

Creating this feature request based on a conversation with a service team

Related component

Browser Testing in Continuous Integration(CI) Environment

Context

At present, our CI Environment supports testing only with two browsers - Chrome and Firefox. There is no option to test cross browser testing with tools like BrowserStack. One of the service teams, Payments, would like to have this feature in the CI environment.

Below are the responses from the Payments Team QA.

Why would they like to run in the CI environment?

It’s because we have a stride service which means hmrc staff will always be on Windows & Edge or IE. Since we most commonly test on Mac & Chrome, it’s not as much reassurance as testing with Windows.

How often would they be running these tests?

It would be nice to have it automated on Jenkins. I’m working with another QA on Payments and he’s got 2 repos set up with browserstack running locally. I’ve got 1 set of tests using it so far, possibly another one could use browserstack too as those users have to switch to a mobile browser.

Alternatives

They are sticking to running locally on their machine.

Additional information

webdriver-factory has an option to test with BrowserStack, but this assumes the necessary BrowserStack connectivity is already in place. So further investigation is required to use this in CI environment.

[JamieNorth]

This would be very helpful as we could trigger these to run before each release we put out, this will provide confidence that users across different devices have a good experience. We can make use of our Test Job viewer too to give quick feedback of any issues found in the test cycle that could be specific to a device/browser. Currently, execution of browser tests are manual and are not run regularly so it would be good to improve this. We have a new service going live in the summer (self-assessment-refund), so implementing something like this would be great to maximise the use of dev/test time and provide a good degree of confidence that the Service will be compliant with a range of browsers.

[Stephen-Kam]

Being able to run browserstack tests as part of a pipeline or at least as a downstream job would be beneficial to providing further confidence in our frontend pages. As GOV UK mandate that their services run on specific browsers and devices, being able to run these cross browser tests would allows us to ensure we're meeting those requirements.

Currently it is essentially a manual process, as someone needs to kick off the tests from their local repo. There's also the matter of ensuring a local instance of Browserstack being set up too which not everyone or many people would be able to do quickly. It would be handy to have this automated in Jenkins, so anyone can run them.

[Kavitakum]

For many projects in HMRC, its required to test the service in multiple browsers and devices. Though it is not done very frequently, but atleast after any major changes and before release, we need to make sure that service is compliant with range of browsers.

Currently, we are using the browserstack for cross-browser testing, but the automation test cases need to be triggered from our local machine and we need to make sure that local instance of browserstack is set up in our machine. This creates a dependency.

Having the browserstack tests run as part of pipeline or as downstream job will provide an option to trigger the testing easily. There would not be any dependency and anyone in team would be able to do it easily. We will get the confidence that service is compliant with multiple browsers and devices. This will also allow anyone to easily check the previous execution results as well via Jenkins.

[davidmthornton]

Having saved a lot of time using accessibility job on the different services I've worked on, I support the idea of having something similar for cross-browser and potential cross-device testing.

There are a number of barriers which could prevent teams from performing high quality compatibility testing including:-

• the tooling setup, development and licensing issue
• continuous release of new browser versions and devices which may or may not be supported
• confusion about which browsers are supported and which devices
• time constraints on the project which may reduce the scope of browser / devices tested
• access to different hardware, browsers (if a cloud solution isn't used such as Browserstack)

The accessibility job provides a list of issues to be resolved, picked up as part of an automated test run. I wonder if the Browserstack job for compatibility testing would be able to do the same or if it would instead provide a set of screenshots and / or failed AT scenarios?

If the Browserstack job is able to report failing test scenarios and which browser / device was affected, it would be enough for the team to be aware of an issue, investigate and resolve it whilst ensuring that the resolution didn't cause problems for other browsers (again as part of the pipeline). If screenshots can accompany the test failures, even more time could potentially be saved.

[ellamdav]

Closing this issue as we're archiving this repo. We’re planning to review the platform's cross browser testing approach within the next few months, and we’ll be reaching out to teams for their feedback then.