microsoft / playwright

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
https://playwright.dev
Apache License 2.0
67.12k stars 3.69k forks source link

[Feature]: Make it possible to set up 'maxFailures' within the project in the config file #33665

Open dmrock opened 3 days ago

dmrock commented 3 days ago

🚀 Feature Request

Make it possible to set up 'maxFailures' within the project in the config file

Here is an example: { name: 'setup', testMatch: /.*\.setup\.ts/, testDir: './tests', maxFailures: 2 },

Example

This setting will allow us to avoid running, for example, all the authorization tests in the setup file used as a dependency for the main project tests.

When this can be useful:

For instance, before testing, we perform 20 authorizations for different user types in a single application. The authorization goes through the UI, and there’s a bug in the UI. If the very first authorization test fails, the main project tests won’t be executed, but the remaining 19 authorization tests will still run, which, considering retries, will take a significant amount of time.

Motivation

It would be much more efficient to stop the execution after the first failure and analyze it as quickly as possible. This is especially important on the CI.