Yelp / Testify

A more pythonic testing framework.
Other
308 stars 67 forks source link

interrupted tests apparently lead to success #170

Closed mrtyler closed 9 years ago

mrtyler commented 11 years ago

Consider this test case from workflow:

14:33:58 tests.bb_run_test DetermineBuildbotTestCase.test_given_valid_buildbot_in_keys ... Invalid buildbot "t" (normalized to 't')- aborting.
14:33:58 INTERRUPTED in 0.00s
14:33:58 
14:33:58 ========================================================================
14:33:58 FAILURES
14:33:58 None!
14:33:58 
14:33:58 FAILED.  22 tests / 5 cases: 21 passed, 0 failed.  (Total test time 0.03s)

Jenkins think this suite passed because Testify returns code 0, but obviously that is not correct. At minimum, we need to correctly report this as an error.

It may also be worth looking at the status string "FAILED. 22 tests / 5 cases: 21 passed, 0 failed." since it contains several lies.

asottile commented 9 years ago

Still a bug afaict:

$ cat test2.py
import testify as T

class F(T.TestCase):
    def test(self):
        raise KeyboardInterrupt
$ testify test2.py
-
FAILED.  1 test / 1 case: 0 passed, 0 failed.  (Total test time 0.00s)
$ echo $?
0