search

kevlened / pytest-parallel

A pytest plugin for parallel and concurrent testing
https://github.com/browsertron/pytest-parallel/issues/104#issuecomment-1293941066
MIT License
313 stars 59 forks source link

Number of tests fail with pytest6 #91

Open mcepl opened 3 years ago

mcepl commented 3 years ago

When building pytest-parallel package for openSUSE/Tumbleweed we found out that number of tests fail with pytest6 (without respect whether we use python 3.6 or 3.8). We had to limit ourselves to pytest5 and then the test suite passes without problems.

[   52s] =========================== short test summary info ============================
[   52s] ERROR tests/test_general.py::test_environ_shim[cli_args2] - AssertionError
[   52s] ERROR tests/test_general.py::test_skip_markers[cli_args0] - AssertionError
[   52s] ERROR tests/test_general.py::test_skip_markers[cli_args1] - AssertionError
[   52s] ERROR tests/test_general.py::test_skip_markers[cli_args2] - AssertionError
[   52s] ERROR tests/test_general.py::test_skipif_markers[cli_args0] - AssertionError
[   52s] ERROR tests/test_general.py::test_skipif_markers[cli_args1] - AssertionError
[   52s] ERROR tests/test_general.py::test_skipif_markers[cli_args2] - AssertionError
[   52s] ERROR tests/test_general.py::test_custom_markers[cli_args0] - AssertionError
[   52s] ERROR tests/test_general.py::test_custom_markers[cli_args1] - AssertionError
[   52s] ERROR tests/test_general.py::test_custom_markers[cli_args2] - AssertionError
[   52s] ERROR tests/test_general.py::test_multiple_failures[cli_args0] - AssertionError
[   52s] ERROR tests/test_general.py::test_multiple_failures[cli_args1] - AssertionError
[   52s] ERROR tests/test_general.py::test_multiple_failures[cli_args2] - AssertionError
[   52s] ERROR tests/test_general.py::test_pytest_html[cli_args0] - AssertionError
[   52s] ERROR tests/test_general.py::test_pytest_html[cli_args1] - AssertionError
[   52s] ERROR tests/test_general.py::test_pytest_html[cli_args2] - AssertionError
[   52s] ERROR tests/test_general.py::test_collection_error[cli_args0] - AssertionError
[   52s] ERROR tests/test_general.py::test_collection_error[cli_args1] - AssertionError
[   52s] ERROR tests/test_general.py::test_collection_error[cli_args2] - AssertionError
[   52s] ERROR tests/test_general.py::test_collection_collectonly[cli_args0] - Asserti...
[   52s] ERROR tests/test_general.py::test_collection_collectonly[cli_args1] - Asserti...
[   52s] ERROR tests/test_general.py::test_collection_collectonly[cli_args2] - Asserti...
[   52s] ======================== 8 passed, 22 errors in 17.99s =========================

Complete build log with details of used packages, their versions, and steps taken for building the package.

mcepl commented 2 years ago

Specifically, one such error looks like this:

[   52s] ___________ ERROR at setup of test_collection_collectonly[cli_args2] ___________
[   52s] 
[   52s] cls = _pytest.runner.CallInfo
[   52s] func = <function call_runtest_hook.<locals>.<lambda> at 0x7f14d7c34c80>
[   52s] when = 'setup'
[   52s] reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
[   52s] 
[   52s]     @classmethod
[   52s]     def from_call(
[   52s]         cls,
[   52s]         func: "Callable[[], TResult]",
[   52s]         when: "Literal['collect', 'setup', 'call', 'teardown']",
[   52s]         reraise: Optional[
[   52s]             Union[Type[BaseException], Tuple[Type[BaseException], ...]]
[   52s]         ] = None,
[   52s]     ) -> "CallInfo[TResult]":
[   52s]         excinfo = None
[   52s]         start = timing.time()
[   52s]         precise_start = timing.perf_counter()
[   52s]         try:
[   52s] >           result: Optional[TResult] = func()
[   52s] 
[   52s] /usr/lib/python3.6/site-packages/_pytest/runner.py:311: 
[   52s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   52s] /usr/lib/python3.6/site-packages/_pytest/runner.py:255: in <lambda>
[   52s]     lambda: ihook(item=item, **kwds), when=when, reraise=reraise
[   52s] /usr/lib/python3.6/site-packages/pluggy/hooks.py:286: in __call__
[   52s]     return self._hookexec(self, self.get_hookimpls(), kwargs)
[   52s] /usr/lib/python3.6/site-packages/pluggy/manager.py:93: in _hookexec
[   52s]     return self._inner_hookexec(hook, methods, kwargs)
[   52s] /usr/lib/python3.6/site-packages/pluggy/manager.py:87: in <lambda>
[   52s]     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
[   52s] /usr/lib/python3.6/site-packages/_pytest/runner.py:150: in pytest_runtest_setup
[   52s]     item.session._setupstate.prepare(item)
[   52s] /usr/lib/python3.6/site-packages/_pytest/runner.py:452: in prepare
[   52s]     raise e
[   52s] /usr/lib/python3.6/site-packages/_pytest/runner.py:449: in prepare
[   52s]     col.setup()
[   52s] /usr/lib/python3.6/site-packages/_pytest/python.py:1647: in setup
[   52s]     self._request._fillfixtures()
[   52s] /usr/lib/python3.6/site-packages/_pytest/fixtures.py:568: in _fillfixtures
[   52s]     item.funcargs[argname] = self.getfixturevalue(argname)
[   52s] /usr/lib/python3.6/site-packages/_pytest/fixtures.py:581: in getfixturevalue
[   52s]     fixturedef = self._get_active_fixturedef(argname)
[   52s] /usr/lib/python3.6/site-packages/_pytest/fixtures.py:601: in _get_active_fixturedef
[   52s]     self._compute_fixture_value(fixturedef)
[   52s] /usr/lib/python3.6/site-packages/_pytest/fixtures.py:687: in _compute_fixture_value
[   52s]     fixturedef.execute(request=subrequest)
[   52s] /usr/lib/python3.6/site-packages/_pytest/fixtures.py:1048: in execute
[   52s]     fixturedef = request._get_active_fixturedef(argname)
[   52s] /usr/lib/python3.6/site-packages/_pytest/fixtures.py:601: in _get_active_fixturedef
[   52s]     self._compute_fixture_value(fixturedef)
[   52s] /usr/lib/python3.6/site-packages/_pytest/fixtures.py:687: in _compute_fixture_value
[   52s]     fixturedef.execute(request=subrequest)
[   52s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   52s] 
[   52s] self = <FixtureDef argname='pytester' scope='function' baseid=''>
[   52s] request = <SubRequest 'pytester' for <Function test_collection_collectonly[cli_args2]>>
[   52s] 
[   52s]     def execute(self, request: SubRequest) -> _FixtureValue:
[   52s]         # Get required arguments and register our own finish()
[   52s]         # with their finalization.
[   52s]         for argname in self.argnames:
[   52s]             fixturedef = request._get_active_fixturedef(argname)
[   52s]             if argname != "request":
[   52s]                 # PseudoFixtureDef is only for "request".
[   52s] >               assert isinstance(fixturedef, FixtureDef)
[   52s] E               AssertionError
[   52s] 
[   52s] /usr/lib/python3.6/site-packages/_pytest/fixtures.py:1051: AssertionError