microsoft / playwright

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
https://playwright.dev
Apache License 2.0
65.49k stars 3.57k forks source link

[Feature]: Add all retries as test runs to JUnit report #29446

Open MrGreenTea opened 7 months ago

MrGreenTea commented 7 months ago

🚀 Feature Request

A config option for the junit reporter to add all retries to the output file.

Example

No response

Motivation

We're mostly using the junit reports together with flaky detection to monitor and detect flaky tests.

We currently have to use the retries config option which masks some of the flakiness, because the tests appear as success in the report.

I would love to be able to include the retries in our analysis for a more complete and precise flakiness detection.

thomas-colgrove commented 4 months ago

+1 would love this data to be included. seeing tests that are flaky can be a good indicator of test suite health

ComaToastUK commented 4 months ago

+1 This would be useful in helping us identify flaky tests

alice-ep commented 1 month ago

+1, our team would love this feature. Right now we're sampling test logs occasionally to try and catch flakiness, but it'd be much nicer to be able to follow this as a metric