search

microsoft / playwright

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
https://playwright.dev
Apache License 2.0
66.98k stars 3.68k forks source link

[Feature]: Forced Failure for Dependent Tests in Playwright #32378

Open farazokc opened 2 months ago

farazokc commented 2 months ago

🚀 Feature Request

An option should be added in Playwright to automatically fail dependent tests when their prerequisite tests fail. This feature would ensure that if setup tests fail, any dependent tests are also reported as failed, rather than skipped.

Currently, in our CI/CD pipelines, setup tests configure portals which are later used by other test projects. If these setup tests fail, dependent tests are skipped, leading to inaccurate test reporting. For example, our pipeline reports show only 10 tests completed out of 200+, as the skipped tests are not properly accounted for.

20240828 1 fake reporting

In the above image, the total tests count shown in 10, even though we have 200+ tests in the pipeline. The rest are skipped as the dependency tests are failed.

Example

No response

Motivation

This feature will help make reporting accurate as both skipped and failed tests will show up as failed, as they are not run in the pipeline.

Abubaker0001 commented 2 months ago

I would like to request this feature too as I am facing this issue as well.