search

delfick / alt-pytest-asyncio

An alternative plugin for pytest to make it support async tests and fixtures
https://alt-pytest-asyncio.readthedocs.io
MIT License
27 stars 5 forks source link

Tests failures with Pytest >= 7 #12

Closed felixonmars closed 1 year ago

felixonmars commented 1 year ago 
==================================== ERRORS ====================================
___________________ ERROR collecting tests/test_examples.py ____________________
ImportError while importing test module '/build/python-alt-pytest-asyncio/src/alt-pytest-asyncio-release-0.6.0/tests/test_examples.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_examples.py:3: in <module>
    from _pytest.pytester import Testdir as TD, LineMatcher
E   ImportError: cannot import name 'Testdir' from '_pytest.pytester' (/usr/lib/python3.11/site-packages/_pytest/pytester.py)

Relevant change in pytest 7.0: https://github.com/pytest-dev/pytest/commit/1df28a4450ae6deecd4f94b775a8e03022be4020

I have tried to change it to import Testdir from _pytest.legacypath instead, but fails with another error:

=================================== FAILURES ===================================
_____ test_shows_correctly_for_failing_fixtures[example_fixture_failures] ______
tests/test_examples.py:88: in test_shows_correctly_for_failing_fixtures
    testdir = TD(request, factory)
E   TypeError: Pytester.__init__() missing 1 required positional argument: 'monkeypatch'
_________ test_shows_correctly_for_failing_fixtures[example_timeouts] __________
tests/test_examples.py:88: in test_shows_correctly_for_failing_fixtures
    testdir = TD(request, factory)
E   TypeError: Pytester.__init__() missing 1 required positional argument: 'monkeypatch'

Relevant change in pytest 7.1: https://github.com/pytest-dev/pytest/commit/f943d1944a9ece4fc5b1b3a83046380fd2ff3fc4

And the tests still fail after applying changes for that:

============================= test session starts ==============================
platform linux -- Python 3.10.10, pytest-7.3.0, pluggy-1.0.0
rootdir: /build/python-alt-pytest-asyncio/src/alt-pytest-asyncio-release-0.6.0
configfile: pytest.ini
plugins: alt-pytest-asyncio-0.6.0, noseOfYeti-2.4.0
collected 17 items

tests/test_examples.py FFF
tests/test_override_loop.py .......
tests/test_tcp_one.py .
tests/test_tcp_two.py .
tests/test_works.py .....

=================================== FAILURES ===================================
_____ test_shows_correctly_for_failing_fixtures[example_fixture_failures] ______
/build/python-alt-pytest-asyncio/src/alt-pytest-asyncio-release-0.6.0/tests/test_examples.py:105: in test_shows_correctly_for_failing_fixtures
    matcher.fnmatch_lines(expected.split("\n"))
E   Failed: exact match: '==================================== ERRORS ===================================='
E   exact match: '_______________ ERROR at setup of test_fails_on_fixture_returns ________________'
E   exact match: 'test_fails.py:16: in fixture_returns'
E   fnmatch: '    await one*()'
E      with: '    await one()'
E   exact match: 'test_fails.py:7: in one'
E   fnmatch: '    return await two*()'
E      with: '    return await two()'
E   exact match: 'test_fails.py:11: in two'
E   fnmatch: '    raise ValueError*("WAT")'
E      with: '    raise ValueError("WAT")'
E   exact match: 'E   ValueError: WAT'
E   exact match: '________________ ERROR at setup of test_fails_on_fixture_yields ________________'
E   exact match: 'test_fails.py:21: in fixture_yields'
E   fnmatch: '    yield await one*()'
E      with: '    yield await one()'
E   exact match: 'test_fails.py:7: in one'
E   fnmatch: '    return await two*()'
E      with: '    return await two()'
E   exact match: 'test_fails.py:11: in two'
E   fnmatch: '    raise ValueError*("WAT")'
E      with: '    raise ValueError("WAT")'
E   exact match: 'E   ValueError: WAT'
E   exact match: '_________ ERROR at teardown of test_fails_on_fixture_fails_in_finally __________'
E   exact match: 'test_fails.py:29: in fixture_fails_in_finally'
E   fnmatch: '    await one*()'
E      with: '    await one()'
E   exact match: 'test_fails.py:7: in one'
E   fnmatch: '    return await two*()'
E      with: '    return await two()'
E   exact match: 'test_fails.py:11: in two'
E   fnmatch: '    raise ValueError*("WAT")'
E      with: '    raise ValueError("WAT")'
E   exact match: 'E   ValueError: WAT'
E   nomatch: '===* 1 passed, 3 error* in * ===*'
E       and: '=============================== warnings summary ==============================='
E       and: '../../../../usr/lib/python3.10/site-packages/_pytest/config/__init__.py:1179'
E       and: '  /usr/lib/python3.10/site-packages/_pytest/config/__init__.py:1179: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: noseOfYeti'
E       and: '    self._mark_plugins_for_rewrite(hook)'
E       and: ''
E       and: '-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html'
E       and: '=========================== short test summary info ============================'
E       and: 'ERROR test_fails.py::test_fails_on_fixture_returns - ValueError: WAT'
E       and: 'ERROR test_fails.py::test_fails_on_fixture_yields - ValueError: WAT'
E       and: 'ERROR test_fails.py::test_fails_on_fixture_fails_in_finally - ValueError: WAT'
E       and: '==================== 1 passed, 1 warning, 3 errors in 0.05s ===================='
E   remains unmatched: '===* 1 passed, 3 error* in * ===*'
----------------------------- Captured stdout call -----------------------------
running: /usr/bin/python -mpytest --basetemp=/tmp/pytest-of-builduser/pytest-0/example_fixture_failures0/runpytest-0 --tb short --basetemp=/tmp/pytest-of-builduser/pytest-0/basetemp
     in: /tmp/pytest-of-builduser/pytest-0/example_fixture_failures0
============================= test session starts ==============================
platform linux -- Python 3.10.10, pytest-7.3.0, pluggy-1.0.0
rootdir: /tmp/pytest-of-builduser/pytest-0/example_fixture_failures0
plugins: alt-pytest-asyncio-0.6.0, noseOfYeti-2.4.0
collected 3 items

test_fails.py EE.E                                                       [100%]

==================================== ERRORS ====================================
_______________ ERROR at setup of test_fails_on_fixture_returns ________________
test_fails.py:16: in fixture_returns
    await one()
test_fails.py:7: in one
    return await two()
test_fails.py:11: in two
    raise ValueError("WAT")
E   ValueError: WAT
________________ ERROR at setup of test_fails_on_fixture_yields ________________
test_fails.py:21: in fixture_yields
    yield await one()
test_fails.py:7: in one
    return await two()
test_fails.py:11: in two
    raise ValueError("WAT")
E   ValueError: WAT
_________ ERROR at teardown of test_fails_on_fixture_fails_in_finally __________
test_fails.py:29: in fixture_fails_in_finally
    await one()
test_fails.py:7: in one
    return await two()
test_fails.py:11: in two
    raise ValueError("WAT")
E   ValueError: WAT
=============================== warnings summary ===============================
../../../../usr/lib/python3.10/site-packages/_pytest/config/__init__.py:1179
  /usr/lib/python3.10/site-packages/_pytest/config/__init__.py:1179: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: noseOfYeti
    self._mark_plugins_for_rewrite(hook)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
ERROR test_fails.py::test_fails_on_fixture_returns - ValueError: WAT
ERROR test_fails.py::test_fails_on_fixture_yields - ValueError: WAT
ERROR test_fails.py::test_fails_on_fixture_fails_in_finally - ValueError: WAT
==================== 1 passed, 1 warning, 3 errors in 0.05s ====================
_________ test_shows_correctly_for_failing_fixtures[example_timeouts] __________
/build/python-alt-pytest-asyncio/src/alt-pytest-asyncio-release-0.6.0/tests/test_examples.py:91: in test_shows_correctly_for_failing_fixtures
    assert not result.errlines
E   assert not ['  + Exception Group Traceback (most recent call last):', '  |   File "/usr/lib/python3.10/runpy.py", line 196, in _r...xec(code, run_globals)', '  |   File "/usr/lib/python3.10/site-packages/pytest/__main__.py", line 5, in <module>', ...]
E    +  where ['  + Exception Group Traceback (most recent call last):', '  |   File "/usr/lib/python3.10/runpy.py", line 196, in _r...xec(code, run_globals)', '  |   File "/usr/lib/python3.10/site-packages/pytest/__main__.py", line 5, in <module>', ...] = <RunResult ret=ExitCode.TESTS_FAILED len(stdout.lines)=65 len(stderr.lines)=83 duration=0.36s>.errlines
----------------------------- Captured stdout call -----------------------------
running: /usr/bin/python -mpytest --basetemp=/tmp/pytest-of-builduser/pytest-0/example_timeouts0/runpytest-0 --tb short --basetemp=/tmp/pytest-of-builduser/pytest-0/basetemp
     in: /tmp/pytest-of-builduser/pytest-0/example_timeouts0
============================= test session starts ==============================
platform linux -- Python 3.10.10, pytest-7.3.0, pluggy-1.0.0
rootdir: /tmp/pytest-of-builduser/pytest-0/example_timeouts0
plugins: alt-pytest-asyncio-0.6.0, noseOfYeti-2.4.0
collected 21 items

test_fails.py .
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/_pytest/main.py", line 323, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/_pytest/main.py", line 348, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/_pytest/runner.py", line 114, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/_pytest/runner.py", line 134, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "teardown", log, nextitem=nextitem))
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/_pytest/runner.py", line 224, in call_and_report
INTERNALERROR>     report: TestReport = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/pluggy/_callers.py", line 55, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/_pytest/skipping.py", line 266, in pytest_runtest_makereport
INTERNALERROR>     rep = outcome.get_result()
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/_pytest/runner.py", line 368, in pytest_runtest_makereport
INTERNALERROR>     return TestReport.from_item_and_call(item, call)
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/_pytest/reports.py", line 365, in from_item_and_call
INTERNALERROR>     longrepr = item._repr_failure_py(
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/_pytest/nodes.py", line 484, in _repr_failure_py
INTERNALERROR>     return excinfo.getrepr(
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/_pytest/_code/code.py", line 671, in getrepr
INTERNALERROR>     return fmt.repr_excinfo(self)
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/_pytest/_code/code.py", line 987, in repr_excinfo
INTERNALERROR>     return ExceptionChainRepr(repr_chain)
INTERNALERROR>   File "/usr/lib/python3.10/site-packages/_pytest/_code/code.py", line 1039, in __init__
INTERNALERROR>     reprtraceback=chain[-1][0],
INTERNALERROR> IndexError: list index out of range
----------------------------- Captured stderr call -----------------------------
  + Exception Group Traceback (most recent call last):
  |   File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
  |     return _run_code(code, main_globals, None,
  |   File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
  |     exec(code, run_globals)
  |   File "/usr/lib/python3.10/site-packages/pytest/__main__.py", line 5, in <module>
  |     raise SystemExit(pytest.console_main())
  |   File "/usr/lib/python3.10/site-packages/_pytest/config/__init__.py", line 189, in console_main
  |     code = main()
  |   File "/usr/lib/python3.10/site-packages/_pytest/config/__init__.py", line 166, in main
  |     ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
  |   File "/usr/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
  |     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  |   File "/usr/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
  |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  |   File "/usr/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
  |     return outcome.get_result()
  |   File "/usr/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
  |     raise ex[1].with_traceback(ex[2])
  |   File "/usr/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
  |     res = hook_impl.function(*args)
  |   File "/usr/lib/python3.10/site-packages/_pytest/main.py", line 316, in pytest_cmdline_main
  |     return wrap_session(config, _main)
  |   File "/usr/lib/python3.10/site-packages/_pytest/main.py", line 304, in wrap_session
  |     config.hook.pytest_sessionfinish(
  |   File "/usr/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
  |     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  |   File "/usr/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
  |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  |   File "/usr/lib/python3.10/site-packages/pluggy/_callers.py", line 55, in _multicall
  |     gen.send(outcome)
  |   File "/usr/lib/python3.10/site-packages/_pytest/terminal.py", line 812, in pytest_sessionfinish
  |     outcome.get_result()
  |   File "/usr/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
  |     raise ex[1].with_traceback(ex[2])
  |   File "/usr/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
  |     res = hook_impl.function(*args)
  |   File "/usr/lib/python3.10/site-packages/_pytest/runner.py", line 108, in pytest_sessionfinish
  |     session._setupstate.teardown_exact(None)
  |   File "/usr/lib/python3.10/site-packages/_pytest/runner.py", line 539, in teardown_exact
  |     raise BaseExceptionGroup("errors during test teardown", exceptions[::-1])
  | exceptiongroup.ExceptionGroup: errors during test teardown (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/usr/lib/python3.10/site-packages/_pytest/runner.py", line 526, in teardown_exact
    |     fin()
    |   File "/usr/lib/python3.10/site-packages/_pytest/fixtures.py", line 686, in <lambda>
    |     subrequest.node.addfinalizer(lambda: fixturedef.finish(request=subrequest))
    |   File "/usr/lib/python3.10/site-packages/_pytest/fixtures.py", line 1038, in finish
    |     raise exc
    |   File "/usr/lib/python3.10/site-packages/_pytest/fixtures.py", line 1031, in finish
    |     func()
    |   File "/build/python-alt-pytest-asyncio/src/alt-pytest-asyncio-release-0.6.0/tmp_install/usr/lib/python3.10/site-packages/alt_pytest_asyncio/async_converters.py", line 208, in finalizer
    |     _run_and_raise(loop, info, generator, async_finalizer())
    |   File "/build/python-alt-pytest-asyncio/src/alt-pytest-asyncio-release-0.6.0/tmp_install/usr/lib/python3.10/site-packages/alt_pytest_asyncio/async_converters.py", line 102, in _run_and_raise
    |     _raise_maybe(func, info)
    |   File "/build/python-alt-pytest-asyncio/src/alt-pytest-asyncio-release-0.6.0/tmp_install/usr/lib/python3.10/site-packages/alt_pytest_asyncio/async_converters.py", line 84, in _raise_maybe
    |     raise_error()
    |   File "/build/python-alt-pytest-asyncio/src/alt-pytest-asyncio-release-0.6.0/tmp_install/usr/lib/python3.10/site-packages/alt_pytest_asyncio/async_converters.py", line 78, in raise_error
    |     assert False, f"Took too long to complete: {fle}:{lineno}"
    | AssertionError: Took too long to complete: /tmp/pytest-of-builduser/pytest-0/example_timeouts0/conftest.py:19
    | assert False
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "/usr/lib/python3.10/site-packages/_pytest/runner.py", line 526, in teardown_exact
    |     fin()
    |   File "/usr/lib/python3.10/site-packages/_pytest/fixtures.py", line 686, in <lambda>
    |     subrequest.node.addfinalizer(lambda: fixturedef.finish(request=subrequest))
    |   File "/usr/lib/python3.10/site-packages/_pytest/fixtures.py", line 1038, in finish
    |     raise exc
    |   File "/usr/lib/python3.10/site-packages/_pytest/fixtures.py", line 1031, in finish
    |     func()
    |   File "/build/python-alt-pytest-asyncio/src/alt-pytest-asyncio-release-0.6.0/tmp_install/usr/lib/python3.10/site-packages/alt_pytest_asyncio/async_converters.py", line 208, in finalizer
    |     _run_and_raise(loop, info, generator, async_finalizer())
    |   File "/build/python-alt-pytest-asyncio/src/alt-pytest-asyncio-release-0.6.0/tmp_install/usr/lib/python3.10/site-packages/alt_pytest_asyncio/async_converters.py", line 102, in _run_and_raise
    |     _raise_maybe(func, info)
    |   File "/build/python-alt-pytest-asyncio/src/alt-pytest-asyncio-release-0.6.0/tmp_install/usr/lib/python3.10/site-packages/alt_pytest_asyncio/async_converters.py", line 84, in _raise_maybe
    |     raise_error()
    |   File "/build/python-alt-pytest-asyncio/src/alt-pytest-asyncio-release-0.6.0/tmp_install/usr/lib/python3.10/site-packages/alt_pytest_asyncio/async_converters.py", line 78, in raise_error
    |     assert False, f"Took too long to complete: {fle}:{lineno}"
    | AssertionError: Took too long to complete: /tmp/pytest-of-builduser/pytest-0/example_timeouts0/test_fails.py:37
    | assert False
    +------------------------------------
__________________ test_cleans_up_tests_properly_on_interrupt __________________
tests/test_examples.py:126: in test_cleans_up_tests_properly_on_interrupt
    p.send_signal(signal.SIGINT)
/usr/lib/python3.10/asyncio/subprocess.py:137: in send_signal
    self._transport.send_signal(signal)
/usr/lib/python3.10/asyncio/base_subprocess.py:145: in send_signal
    self._check_proc()
/usr/lib/python3.10/asyncio/base_subprocess.py:142: in _check_proc
    raise ProcessLookupError()
E   ProcessLookupError
=============================== warnings summary ===============================
../../../../usr/lib/python3.10/site-packages/_pytest/config/__init__.py:1179
  /usr/lib/python3.10/site-packages/_pytest/config/__init__.py:1179: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: noseOfYeti
    self._mark_plugins_for_rewrite(hook)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/test_examples.py::test_shows_correctly_for_failing_fixtures[example_fixture_failures]
FAILED tests/test_examples.py::test_shows_correctly_for_failing_fixtures[example_timeouts]
FAILED tests/test_examples.py::test_cleans_up_tests_properly_on_interrupt - P...
=================== 3 failed, 14 passed, 1 warning in 3.10s ====================
delfick commented 1 year ago

well that was more annoying to fix than I would have liked. I'll make a release later on :)

delfick commented 1 year ago

Version 0.7.0 now released

felixonmars commented 1 year ago

Thanks!

delfick commented 1 year ago

You're welcome :)