[Suggestion] Enhance flaky tests visibility

[Az8th]

Hello,

By looking at the code of this reporter, I found out that a flaky test is actually counted as failed. Meanwhile, Playwright HTML reporter treats them as separated statuses and as we can access this info through the Playwright API, I found it unfortunate to not match the behaviour of the main reporter.

Could we please find a way to mention the number of flaky tests in the card ?

I see several solutions, sorted by increasing complexity level : 1 - Leave as is and mention this behaviour in the docs. 2 - Add the number of flaky tests in paranthesis alongside failed (or passed) tests.

3 - Add a dedicated row, and a third state for card color & text to exisiting ones (passed/failed, with failed being prioritary over flaky)

Note that different mentioning when flaky would be interesting too.

What are you thoughts on this ?

Thanks ;)

[estruyf]

Makes sense, I can add an additional row in the table for flaky tests.

[estruyf]

@Az8th some logic had to be changed, but this could become the result:

[estruyf]

Released in version playwright-msteams-reporter@0.0.8-beta.9974682 - let me know what you think about it.

[Az8th]

Awesome ! What a productivity 😮 Could I suggest to put background color of skipped tests in yellow and flaky in orange ? Or match the colors of the emoji proposition I made in #5 by making skipped ones to blue and flaky in yellow to match HTML reporter ?

[estruyf]

The colors are based on the table column styles which are available from the Adaptive Card.

Currently, it is configured as follows:

• Passed -> Good
• Failed -> Attention
• Skipped -> Accent
• Flaky -> Warning

[estruyf]

Oh @Az8th, just noticed that I shared the old screenshot above. Skipped was as you suggested blue, and skipped in yellow.

[estruyf]

Released in version 0.0.8