Some exceptions get hidden

[beloglazov]

Hi Karol,

I've found another problem :) First, I'll show the output from nosetests without pyqcy:

from mocktest import *

def some_function(obj):
    raise Exception('Some random exception')
    return obj.some_method()

class Mocktest(TestCase):

    def test(self):
        some_object = mock('some object')
        expect(some_object).some_method().and_return(True).once()
        assert some_function(some_object)

Running this test with nosetests results in the following error (the module is called test_mocktest):

======================================================================
ERROR: test (tests.test_mocktest.Mocktest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/anton/repos/openstack-neat/tests/test_mocktest.py", line 15, in test
    assert some_function(some_object)
  File "/home/anton/repos/openstack-neat/tests/test_mocktest.py", line 5, in some_function
    raise Exception('Some random exception')
Exception: Some random exception

======================================================================
FAIL: test (tests.test_mocktest.Mocktest)
----------------------------------------------------------------------
AssertionError: Mock "some_method" did not match expectations:
 expected exactly 1 calls with arguments equal to: ()
 received 0 calls

----------------------------------------------------------------------
Ran 32 tests in 2.408s

FAILED (errors=1, failures=1)

The output is correct: it shows both the exception raised by some_function and by mocktest. Now, let's do the same but with pyqcy:

from mocktest import *
from pyqcy import *

def some_function(obj):
    raise Exception('Some random exception')
    return obj.some_method()

class Mocktest(TestCase):

    @qc(1)
    def test():
        with MockTransaction:
            some_object = mock('some object')
            expect(some_object).some_method().and_return(True).once()
            assert some_function(some_object)

The output is following:

======================================================================
ERROR: [pyqcy] test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pyqcy/integration.py", line 60, in test
    run_tests([prop], verbosity=0, propagate_exc=True)
  File "/usr/lib/python2.7/site-packages/pyqcy/runner.py", line 68, in run_tests
    failure.propagate_failure()
  File "/usr/lib/python2.7/site-packages/pyqcy/properties.py", line 127, in test_one
    result.tags = self.__execute_test(coroutine)
  File "/usr/lib/python2.7/site-packages/pyqcy/properties.py", line 148, in __execute_test
    obj = next(coroutine)
  File "/usr/lib/python2.7/site-packages/pyqcy/properties.py", line 81, in generator_func
    func(*args, **kwargs)
  File "/home/anton/repos/openstack-neat/tests/test_mocktest.py", line 18, in test
    assert some_function(some_object)
  File "/usr/lib/python2.7/site-packages/mocktest/transaction.py", line 39, in __exit__
    raise errors[0]
CheckError: test failed due to AssertionError: Mock "some_method" did not match expectations:
 expected exactly 1 calls with arguments equal to: ()
 received 0 calls
Failure encountered for data:

----------------------------------------------------------------------
Ran 32 tests in 2.438s

FAILED (errors=1)

As you can see, the exception raised by mocktest has overriden the 'Some random exception'. If we remove the exception by mocktest by changing expect to when, the hidden exception shows up:

from mocktest import *
from pyqcy import *

def some_function(obj):
    raise Exception('Some random exception')
    return obj.some_method()

class Mocktest(TestCase):

    @qc(1)
    def test():
        with MockTransaction:
            some_object = mock('some object')
            when(some_object).some_method().and_return(True)
            assert some_function(some_object)

The output is:

======================================================================
ERROR: [pyqcy] test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pyqcy/integration.py", line 60, in test
    run_tests([prop], verbosity=0, propagate_exc=True)
  File "/usr/lib/python2.7/site-packages/pyqcy/runner.py", line 68, in run_tests
    failure.propagate_failure()
  File "/usr/lib/python2.7/site-packages/pyqcy/properties.py", line 127, in test_one
    result.tags = self.__execute_test(coroutine)
  File "/usr/lib/python2.7/site-packages/pyqcy/properties.py", line 148, in __execute_test
    obj = next(coroutine)
  File "/usr/lib/python2.7/site-packages/pyqcy/properties.py", line 81, in generator_func
    func(*args, **kwargs)
  File "/home/anton/repos/openstack-neat/tests/test_mocktest.py", line 18, in test
    assert some_function(some_object)
  File "/home/anton/repos/openstack-neat/tests/test_mocktest.py", line 6, in some_function
    raise Exception('Some random exception')
CheckError: test failed due to Exception: Some random exception
Failure encountered for data:

----------------------------------------------------------------------
Ran 32 tests in 2.405s

FAILED (errors=1)

This problem significantly complicates debugging in some cases. Is there any way to fix this (display all exceptions)?

Thanks, Anton

[Xion]

I'm not sure what causes this, but I'd wager it's because of the slightly different way the unittest runner handles failed assertions (AssertionError) and other types of exceptions. The former are displayed as failures (F), while the latter are errors (E).

I just looked up and it turns out that pyqcy.CheckError inherits directly from Exception; I wonder whether changing the base to AssertionError would change the behavior in desired direction. Even it won't, it's probably a good idea anyway, for the check errors are indeed a form of assertion errors; if it helps Python test runner to handle them better, that's win-win :)

[beloglazov]

Sounds good! I'm looking forward to testing the update.

[beloglazov]

Hi Karol, are you going to make the change you mentioned, or do you want me to test it first?

[Xion]

Sorry, I had some other stuff to do over the weekend :)

Nevertheless, I checked my hypothesis and it seems nothing really changed, except for error changing to failure which is pretty much expected:

F
======================================================================
FAIL: [pyqcy] test
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/karol/own/pyqcy/pyqcy/integration.py", line 60, in test
    run_tests([prop], verbosity=0, propagate_exc=True)
  File "/home/karol/own/pyqcy/pyqcy/runner.py", line 68, in run_tests
    failure.propagate_failure()
  File "/home/karol/own/pyqcy/pyqcy/properties.py", line 127, in test_one
    result.tags = self.__execute_test(coroutine)
  File "/home/karol/own/pyqcy/pyqcy/properties.py", line 148, in __execute_test
    obj = next(coroutine)
  File "/home/karol/own/pyqcy/pyqcy/properties.py", line 81, in generator_func
    func(*args, **kwargs)
  File "/home/karol/own/pyqcy/issue25.py", line 17, in test
    assert some_function(some_object)
  File "/home/karol/own/pyqcy/issue25.py", line 6, in some_function
    raise Exception('Some random exception')
CheckError: test failed due to Exception: Some random exception

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)

The reason you are seeing two exceptions in the first case is, I think, because of the fact that mocktest's AssertionError (the one about failed expectations) is raised in the test's tearDown and thus runner can intercept it separately. With pyqcy and with MockTransaction, mocktest's AssertionError would have to be raised while still in the test. However, this would have to happen while handling the exception for some_function, because this is what causes the with block to exit (somewhat prematurely). Because Python preserves only one exception context, one of those two will be lost.

My latest idea is to fiddle with run method of TestCase, and also look into how mocktest handles all this stuff inside its own TestCase. I hope some approach will bear some fruit finally :)

[Xion]

Okay, I'm confused now.

I scoured the code of mocktest's TestCase and found out that it doesn't really do anything in run method which would be able to produce both an error and a failure for a single test. (It only turns its own exceptions - those about failed expectations - into failures, analogously how I do in pyqcy now but in more complicated way). I even tried to incorporate a similar version of run into pyqcy TestCase; nothing changed.

So, I grew somewhat suspicious and ran your original code, without pyqcy at all:

from mocktest import *

def some_function(obj):
    raise Exception('Some random exception')
    return obj.some_method()

class Mocktest(TestCase):

    def test(self):
        some_object = mock('some object')
        when(some_object).some_method().and_return(True)
        assert some_function(some_object)

This is what I got:

E
======================================================================
ERROR: test (issue25.Mocktest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/karol/own/pyqcy/issue25.py", line 19, in test
    assert some_function(some_object)
  File "/home/karol/own/pyqcy/issue25.py", line 9, in some_function
    raise Exception('Some random exception')
Exception: Some random exception

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (errors=1)

Just the exception from some_function, no mocktest failed expectations' error. Clearly a different result that yours.

mocktest is freshly installed in newest version ($ pip install mocktest). Something is still off there, though. I noticed that your first listing says 32 tests, though; are you sure your don't run your Mocktest test case twice, or something? :)

[beloglazov]

To get that mocktest's expectation exception, you need to use:

expect(some_object).some_method().and_return(True).once()

instead of:

when(some_object).some_method().and_return(True)

Don't worry about 32 tests, that's just because this test file was in the same directory where my project's other tests are. So when I used nosetests, it ran all of them :)