no guarantee that all teardowns are run

[blampe]

import testify as T

class MyTestCase(T.TestCase):

        def test_something(self):
                print 'test called'
                assert False

        @T.teardown
        def teardown(self):
                print 'teardown called'
                assert False

        @T.teardown
        def teardown2(self):
                print 'teardown2 called'
                assert False

Running this prints the following:

test called
teardown called
fail: my_test MyTestCase.test_something
Traceback (most recent call last):
  File "./my_test.py", line 14, in teardown
    assert False
AssertionError

F
FAILED.  1 test / 1 case: 0 passed, 1 failed.  (Total test time 0.00s)

This means it's possible to not rollback important work, which can cause cascading failures.

[sumeet]

testify prints the failure message from the first teardown that failed instead of the failing test. i prefer rspec's behavior of printing out the first exception. in other words, preferring the failure from the test over the failure from the teardown.

describe "blah" do
  after(:each) do
    p "ran after #1"
    fail
  end

  after(:each) do
    p "ran after #2"
    fail
  end

  it "raises an error" do
    fail "failed inside of test"
  end
end

$ rspec test_spec.rb
"ran after #2"
"ran after #1"
F

Failures:

  1) blah raises an error
     Failure/Error: fail "failed inside of test"
     RuntimeError:
       failed inside of test
     # ./test_spec.rb:13:in `block (2 levels) in <top (required)>'

Finished in 0.00139 seconds
1 example, 1 failure

Failed examples:

rspec ./test_spec.rb:12 # blah raises an error

[mrtyler]

@sumeet, if I'm understanding correctly, rspec basically swallows the failures in the teardown methods? Testify used to do that and it was terrible.

[sumeet]

that's not what i'm talking about. it's confusing because there are two separate issues in testify i want to fix.

part 1: i want testify to run all teardown methods

class MyTest(TestCase):

    @teardown
    def teardown_1(self):
        print 'teardown_1'
        raise Teardown1Error

    @teardown
    def teardown_2(self):
        print 'teardown_2'
        raise Teardown2Error

    def test_something(self):
        raise TestError

before testify has a chance to print the error message, testify will only print

teardown_1

because it'll only try to run the first teardown. so

1) testify will also try to run at least one teardown if the test fails 2) testify only runs the first teardown that fails

i'd rather see behavior closer to junit or rspec. i mention junit and rspec because they have exactly the same behavior, and it's the behavior i wanted after getting bit by this a couple of times. i reckon they ran into the same issue and reached the same conclusion. they behave like this:

1) run all teardown methods whether or not the test passed

instead, running the test should print out

teardown_1
teardown_2

because all teardown methods were ran.

part 2: which error gets shown

check out the same test class again

in the current version of testify, if you ran this test, the failure message would be the following:

error: blah MyTest.test_something
Traceback (most recent call last):
  File "./blah.py", line 14, in teardown_1
    raise Teardown1Error
Teardown1Error

this doesn't show you what the failure message from the test was. i'd rather see the stack trace from the first thing that failed. the first thing that failed could be a test, or it could be a teardown action.

if you ran the example above, it'd be more useful to see

Traceback (most recent call last):
  File "./blah.py", line 22, in test_something
    raise TestError
TestError

or if you ran this example

class MyTest(TestCase):

    @teardown
    def teardown_1(self):
        print 'teardown_1'
        raise Teardown1Error

    @teardown
    def teardown_2(self):
        print 'teardown_2'
        raise Teardown2Error

    def test_something(self):
        print 'test passes'

you'd wanna see

Traceback (most recent call last):
  File "./blah.py", line 14, in teardown_1
    raise Teardown1Error
Teardown1Error

or you could show all of them. most importantly, don't show a teardown that failed instead of the test failure.

[baris]

I'll take this issue to make testify run all teardown methods. Sumeet please create a separate issue for the other issue you write about.

[baris]

Pull request #135 fixed the originally reported issue. I'll let Sumeet to create a separate issue for the second issue he mentioned and close this issue now.