Making 'flaky' tests appear as 'failures'

What would you like?

Hello all,

Currently, our team looks at our automation results daily. Whenever the team looks at something that looks like this:

Then the team concludes that everything is fine and we have a "100% passing run" and then we move on.

Whenever the team sees something like this:

Then the team is disappointed about the fact that one of the tests failed and we look into why the test failed.

As you can see, the fact that the "flaky" test still shows as a "passing" test in the first set of results gives in the perception that things are good and we don't end up looking into the failures for the flaky tests.

We would like to enable a method to report a "flaky" test as a "failed" and/or a "flaky" test (instead of a "Pass") while having retries enabled.

We will be able to see (upon clicking into the run details) that the test did pass on retry which is enough to tell us that it is a flaky test and enough for us to avoid leaving things "as-is" because the perception of having a "fully passing run"

Why is this needed?

This is needed to ensure that a "flaky" test is not dismissed so easily. This will help avoid building a backlog of flaky tests and also avoid potential bugs from escaping because of a passing retry giving the perception that the tests are all passing.

Other

No response

cypress-io / cypress