[Proposal] Cross browser testing

getgauge / gauge

Light weight cross-platform test automation

https://gauge.org

Apache License 2.0

2.99k stars 344 forks source link

[Proposal] Cross browser testing #510

Closed sswaroopgupta closed 4 years ago

sswaroopgupta commented 8 years ago

Looking for cross browser testing features in Gauge? Here is a scenario with basic set of requirements and some open questions. Please help us to understand your requirements better.

Scenario: As a Gauge user, I have an automation test suite in Gauge for testing my SUT(System Under Test), a web application. I want to test it against multiple browsers.

Requirements

run tests against multiple browsers say, FF, Chrome with supported OS in my local box.
run on IE with Windows in my local box.
run tests in multiple platforms.
run these tests in parallel.
do you have other requirements?

Open Questions

How do you want to run test suite in multiple platforms? Would you like to use a grid/docker/any other options?
As a user, how would you like to configure it?
What kind of output do you prefer?
Will it be helpful to see comparable outputs in reports for specs run in different platform/browser?
What kind of comparable statistics would you need?

steve-gray commented 8 years ago

@sguptatw:

I think you're asking the question here pretty much the wrong way. If people are after a browser testing thing - that's easily doable today (Selenium SDK wired up to step implementatons) - and building it into the core of Gauge seems to pollute the concept a little.

The interesting part of what you are proposing is the ability to run multiple permutations of the same scenarios but with a varied matrix of inputs, similiar to how Drone.io 0.4/0.5 does permutations. This has uses far beyond just web testing, such as multi-device testing using Appium, or cross-testing an application against multiple versions of various dependencies (i.e. Does my app work on MongoDB 1/2/3/4 versions?).

I'm acheving this effect today with GoCD by using multiple agents with different capabilities set in an environment and instructing a gauge-job to run on all agents - but supporting matrix-driven execution at a suite level would be a majorly useful element.

apoorvam commented 8 years ago

@steve-gray Totally agree with your opinion.

Let me clarify few points in this proposal.

This feature aims at executing test suite against multiple environments, leveraging the concept of environment that Gauge already has. Tests could be executed against multiple environments in parallel too.
Reports would consolidate the execution results from all environments and show statistics that helps in comparing.
This can be used to perform cross-browser testing, multi-device testing, testing across multiple versions of applications/dependencies. Cross browser testing is one potential use case for this feature.
Although Gauge could enable multiple environments, the onus is on the implemented code to benefit from this, i.e. Gauge shall not attempt to handle/resolve any contentions that may come up at code level e.g. race conditions, browser specific configs.

Any further input is most welcome.

sswaroopgupta commented 8 years ago

@steve-gray: Thank you for your feedback and inputs.

zabil commented 4 years ago

Cross browser involves a lot of parameters which are outside Gauge's control like infratructure setup etc. It's best run tests independently and collect the results to build a report than building features into Gauge for handling it.

That said Gauge is an acceptance testing frameworks and the tests need not be cross browser. As mentioned in the comments above this is about managing multiple environments.

For now this feature won't be implemented.