search

allure-framework / allure2

Allure Report is a flexible, lightweight multi-language test reporting tool. It provides clear graphical reports and allows everyone involved in the development process to extract the maximum of information from the everyday testing process
https://allurereport.org/
Apache License 2.0
4.2k stars 712 forks source link

Flaky tests #111

Closed baev closed 7 years ago

thasherwin commented 7 years ago

Is it a good idea to have a color for flaky tests in the graph section? We do see that a test is marked as flaky with the icon. But looking on the graph, you will not know how many failed as flaky.

baev commented 7 years ago

sounds good

https://github.com/allure-framework/allure2/issues/171

just-boris commented 7 years ago

How do you see this, guys?

Currently we have 4 statues with four different colors. How should we do to display flacky test? It can also be either passed or not passed. Threoretically, it means that we need to create a color for flacky-passed and flacky-failed and flacky-broken tests. It will make the chart unreadable.

I'd suggest to add an extra chart that displays only pure statistics, excluding tests, that we considered flacky. Aslo it's possible to add a checkbox like "Show flacky tests", that will switch the mode for the graph.

sseliverstov commented 7 years ago

How about dashes? white(transparent) + color of current status?

baev commented 7 years ago

Note that we have flaky, muted and known errors now. And I agree with @just-boris that different color for every case will make charts unreadable.

Also we are going to introduce a categories in forthcoming release. So every error may have (or have not) a category such as infrastructure, network and so on. And every category will have its own color as well.

I propose to discuss how we gonna achieve this in https://github.com/allure-framework/allure2/issues/171