search

microsoft / playwright

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
https://playwright.dev
Apache License 2.0
65.74k stars 3.58k forks source link

[Feature]: Fail after all tests failure percent #32690

Open vim-daniel opened 1 day ago

vim-daniel commented 1 day ago

🚀 Feature Request

In the test configurations add a property abandonAfterFailuresPercentage When a test fails we should check if the sum of already failed tests pass this percentage in such a case we should not continue running more tests and instead finish the test run with the already failed tests results.

Test retries should not be counted, meaning text X failed and after 2 retries it kept failing - it should be counted as 1 failure. also if test Y failed and after 2 retries it passed - it should not count as a failure

Example

No response

Motivation

The motivation for this is to fail fast - usually when you have many tests and a bunch of them fail it's because of a common issue, in which case waiting for the full test run & retries can be very time consuming and not helpful.

mxschmitt commented 1 day ago

Playwright has --max-failures=123 where you can specify that. In the next Playwright version it doesn't include retries anymore. So I guess the feature request could be about supporting percentage in this CLI option?

(when talking about CLI option I also mean config option)