Yelp / Testify

A more pythonic testing framework.
Other
308 stars 67 forks source link

Teardown Exceptions Mask Test Exceptions. #164

Closed kmosher closed 11 years ago

kmosher commented 11 years ago

If a test raises an exception, then the teardown gets run. If that also raises an exception, then the only exception returned will be the one thrown in the teardown.

Teardown exceptions should be caught, printed, and the original test exception re-raised.

eevee commented 11 years ago

They also mask setup exceptions, which is doubly bonkers since an exception in setup makes it highly likely that teardown will fail.

mrtyler commented 11 years ago

175 was a dupe of this. Here's what @bukzor said over there:

When a test has two failures, the first failure is nearly always the most relevant.

As currently implemented, when a test fails, then its teardown fails, only the teardown failure is shown by testify.

An improvement would be to only show the original failure. Ideally we could show both failures (and label them accordingly).

In terms of priority, this makes debugging failures which incidentally cause teardown failures nearly impossible. The only way to proceed in that case is to disable the offending teardown process entirely, and run the test again.

bukzor commented 11 years ago

I believe this issue is what's making it so hard for us to find the cause of our current selenium/saucelabs issues.

I posit that this is a teardown error that's masking the root-cause error: http://pb/55601