search

pytest-dev / pytest-qt

pytest plugin for Qt (PyQt5/PyQt6 and PySide2/PySide6) application testing
https://pytest-qt.readthedocs.io
MIT License
409 stars 70 forks source link

Logging hookwrapper hides exceptions #98

Closed The-Compiler closed 9 years ago

The-Compiler commented 9 years ago

I'm currently investigating a problem with pytest-bdd where it raises an INTERNALERROR> - unfortunately, pytest-qt hides it with another one :wink:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/main.py", line 90, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/main.py", line 121, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 724, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 333, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 596, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/main.py", line 146, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 724, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 333, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 595, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 253, in _wrapped_call
INTERNALERROR>     return call_outcome.get_result()
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 278, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 264, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 595, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 253, in _wrapped_call
INTERNALERROR>     return call_outcome.get_result()
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 278, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 264, in __init__
INTERNALERROR>     self.result = func()
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 596, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/runner.py", line 65, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/runner.py", line 75, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/runner.py", line 121, in call_and_report
INTERNALERROR>     report = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 724, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 333, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 595, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 249, in _wrapped_call
INTERNALERROR>     wrap_controller.send(call_outcome)
INTERNALERROR>   File "/home/florian/proj/qutebrowser/git/.tox/py35/lib/python3.5/site-packages/pytestqt/logging.py", line 43, in pytest_runtest_makereport
INTERNALERROR>     report = outcome.result
INTERNALERROR> AttributeError: '_CallOutcome' object has no attribute 'result'

Looking at it with pdb, that seems to be the case because there was an exception:

(Pdb) outcome.result
*** AttributeError: '_CallOutcome' object has no attribute 'result'
(Pdb) outcome.excinfo
(<class 'IndexError'>, IndexError('list index out of range',), <traceback object at 0x7f1645d85688>)

Shouldn't pytest-qt use outcome.get_result() instead which raises the exception if there is any?

nicoddemus commented 9 years ago

Hmm you're right, it should use outcome.get_result(). Thanks! :smile:

The-Compiler commented 9 years ago

I tried writing a test for this:

def test_logging_broken_makereport(testdir):
    """
    Make sure logging's makereport hookwrapper doesn't hide exceptions.

    See https://github.com/pytest-dev/pytest-qt/issues/98

    :type testdir: _pytest.pytester.TmpTestdir
    """
    testdir.makepyfile(
    conftest="""
        import pytest

        @pytest.mark.hookwrapper(tryfirst=True)
        def pytest_runtest_makereport():
            raise Exception("This should not be hidden")
    """)
    testdir.makepyfile(
        """
        def test_foo():
            pass
        """
    )
    res = testdir.runpytest()
    res.stdout.fnmatch_lines([
        '*This should not be hidden*',
    ])

but it passes...

The error I got with pytest-bdd in the hidden exception was https://github.com/pytest-dev/pytest-bdd/issues/155 (which is probably hiding another exception... :scream:)

Maybe this should just be fixed without a regression test because it might be difficult to get one? :wink:

nicoddemus commented 9 years ago

Thanks for trying! I will give it a shot myself, but I agree that it should just be fixed even if we can't find a proper test for it.

nicoddemus commented 9 years ago

I was able to modify your test a bit to make the test fail as we expected (27a1aaa). Thanks! :smile: