It would be helpful to have a way to run tests that are known to be flakey but not have the test results be counted towards passing or failing in the final result. It is helpful to run flakey tests to see how often they fail in the context of all the other tests because the flakiness could be caused by the order of the test runs.
The --exclude functionality simply skips the tests and the @ignore decorator also does not run tests but at least mentions in the result tests not run which is helpful to have a quick summary of how many flakey tests were ignored.
In the old issue #112 there was a mention of a decorator called @ok_to_fail that seems to imply the functionality desired. The decorator approach helps keep the expected test behaviour close to the definition of the test.
Would need reporter.py to categorize the results of tests decorated as @ok_to_fail. I suggest having 2 additional test status OPASS and OFAIL to distinguish from existing test status PASS, FAIL, and IGNORE.
It would be helpful to have a way to run tests that are known to be flakey but not have the test results be counted towards passing or failing in the final result. It is helpful to run flakey tests to see how often they fail in the context of all the other tests because the flakiness could be caused by the order of the test runs.
The
--excludefunctionality simply skips the tests and the@ignoredecorator also does not run tests but at least mentions in the result tests not run which is helpful to have a quick summary of how many flakey tests were ignored.In the old issue #112 there was a mention of a decorator called
@ok_to_failthat seems to imply the functionality desired. The decorator approach helps keep the expected test behaviour close to the definition of the test.Would need
reporter.pyto categorize the results of tests decorated as@ok_to_fail. I suggest having 2 additional test statusOPASSandOFAILto distinguish from existing test statusPASS,FAIL, andIGNORE.