search

Yelp / MOE

A global, black box optimization engine for real world metric optimization.
Other
1.31k stars 139 forks source link

[Python] set up a common file for test markers/decorators (e.g., to mark flaky, excluded, slow, etc) #407

Open suntzu86 opened 9 years ago

suntzu86 commented 9 years ago

pytest provides a nice framework for skipping tests. It can handle conditionals, expected failures, and much more: http://pytest.org/latest/skipping.html

e.g., skipping tests that are marked slow is as easy as: http://pytest.org/latest/example/simple.html#control-skipping-of-tests-according-to-command-line-option

The system relies on devs decorating tests consistently with pytest.mark subclasses. They suggest:

For larger test suites it’s usually a good idea to have one file where you define the markers which you then consistently apply throughout your test suite.

We should set up such a file for ourselves. At the minimum it should have a standard way of marking tests as "excluded" so we can quickly disable them. (There's even frameworks for rerunning flaky tests automatically: https://github.com/box/flaky)