Closed blampe closed 11 years ago
testify prints the failure message from the first teardown that failed instead of the failing test. i prefer rspec's behavior of printing out the first exception. in other words, preferring the failure from the test over the failure from the teardown.
describe "blah" do
after(:each) do
p "ran after #1"
fail
end
after(:each) do
p "ran after #2"
fail
end
it "raises an error" do
fail "failed inside of test"
end
end
$ rspec test_spec.rb
"ran after #2"
"ran after #1"
F
Failures:
1) blah raises an error
Failure/Error: fail "failed inside of test"
RuntimeError:
failed inside of test
# ./test_spec.rb:13:in `block (2 levels) in <top (required)>'
Finished in 0.00139 seconds
1 example, 1 failure
Failed examples:
rspec ./test_spec.rb:12 # blah raises an error
@sumeet, if I'm understanding correctly, rspec basically swallows the failures in the teardown methods? Testify used to do that and it was terrible.
that's not what i'm talking about. it's confusing because there are two separate issues in testify i want to fix.
part 1: i want testify to run all teardown methods
class MyTest(TestCase):
@teardown
def teardown_1(self):
print 'teardown_1'
raise Teardown1Error
@teardown
def teardown_2(self):
print 'teardown_2'
raise Teardown2Error
def test_something(self):
raise TestError
before testify has a chance to print the error message, testify will only print
teardown_1
because it'll only try to run the first teardown. so
1) testify will also try to run at least one teardown if the test fails 2) testify only runs the first teardown that fails
i'd rather see behavior closer to junit or rspec. i mention junit and rspec because they have exactly the same behavior, and it's the behavior i wanted after getting bit by this a couple of times. i reckon they ran into the same issue and reached the same conclusion. they behave like this:
1) run all teardown methods whether or not the test passed
instead, running the test should print out
teardown_1
teardown_2
because all teardown methods were ran.
part 2: which error gets shown
check out the same test class again
in the current version of testify, if you ran this test, the failure message would be the following:
error: blah MyTest.test_something
Traceback (most recent call last):
File "./blah.py", line 14, in teardown_1
raise Teardown1Error
Teardown1Error
this doesn't show you what the failure message from the test was. i'd rather see the stack trace from the first thing that failed. the first thing that failed could be a test, or it could be a teardown action.
if you ran the example above, it'd be more useful to see
Traceback (most recent call last):
File "./blah.py", line 22, in test_something
raise TestError
TestError
or if you ran this example
class MyTest(TestCase):
@teardown
def teardown_1(self):
print 'teardown_1'
raise Teardown1Error
@teardown
def teardown_2(self):
print 'teardown_2'
raise Teardown2Error
def test_something(self):
print 'test passes'
you'd wanna see
Traceback (most recent call last):
File "./blah.py", line 14, in teardown_1
raise Teardown1Error
Teardown1Error
or you could show all of them. most importantly, don't show a teardown that failed instead of the test failure.
I'll take this issue to make testify run all teardown methods. Sumeet please create a separate issue for the other issue you write about.
Pull request #135 fixed the originally reported issue. I'll let Sumeet to create a separate issue for the second issue he mentioned and close this issue now.
Running this prints the following:
This means it's possible to not rollback important work, which can cause cascading failures.