3.15.0: pytest fails in 3 units

[kloczek]

Python Version

3.9.18

pytest Version

8.1.1

Package Version

3.15.0

Description

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix> using installer module
• run pytest with $PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>
• build is performed in env which is cut off from access to the public network (pytest is executed with -m "not network")

Here is pytest output:

```console + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pytest-randomly-3.15.0-2.fc36.x86_64/usr/lib64/python3.9/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pytest-randomly-3.15.0-2.fc36.x86_64/usr/lib/python3.9/site-packages + /usr/bin/pytest -ra -m 'not network' ============================= test session starts ============================== platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0 Using --randomly-seed=722815198 rootdir: /home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.15.0 configfile: pyproject.toml plugins: pytest_randomly-3.15.0, Faker-24.2.0, forked-1.6.0, xdist-3.5.0 collected 37 items tests/test_pytest_randomly.py ..F.......F........................F. [100%] =================================== FAILURES =================================== __________________________ test_entrypoint_injection ___________________________ pytester = monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7ff0ff7af400> def test_entrypoint_injection(pytester, monkeypatch): """Test that registered entry points are seeded""" (pytester.path / "test_one.py").write_text("def test_one(): pass\n") class _FakeEntryPoint: """Minimal surface of Entry point API to allow testing""" def __init__(self, name: str, obj: mock.Mock) -> None: self.name = name self._obj = obj def load(self) -> mock.Mock: return self._obj entry_points: list[_FakeEntryPoint] = [] def fake_entry_points(*, group): return entry_points monkeypatch.setattr(pytest_randomly, "entry_points", fake_entry_points) reseed = mock.Mock() entry_points.append(_FakeEntryPoint("test_seeder", reseed)) # Need to run in-process so that monkeypatching works pytester.runpytest_inprocess("--randomly-seed=1") > assert reseed.mock_calls == [ mock.call(1), mock.call(1), mock.call(0), mock.call(1), mock.call(2), ] E assert [] == [call(1), cal...l(1), call(2)] E E Right contains 5 more items, first extra item: call(1) E Use -v to get more diff /home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.15.0/tests/test_pytest_randomly.py:769: AssertionError ----------------------------- Captured stdout call ----------------------------- ============================= test session starts ============================== platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0 Using --randomly-seed=1 rootdir: /tmp/pytest-of-tkloczko/pytest-212/test_entrypoint_injection0 plugins: pytest_randomly-3.15.0, Faker-24.2.0, forked-1.6.0, xdist-3.5.0 collected 1 item test_one.py . [100%] ============================== 1 passed in 0.03s =============================== _________________________ test_it_runs_before_stepwise _________________________ ourtester = def test_it_runs_before_stepwise(ourtester): ourtester.makepyfile( test_one=""" def test_a(): assert 0 def test_b(): assert 0 """ ) out = ourtester.runpytest("-v", "--randomly-seed=1", "--stepwise") out.assert_outcomes(failed=1) # Now make test_b pass ourtester.makepyfile( test_one=""" def test_a(): assert 0 def test_b(): assert 1 """ ) > shutil.rmtree(ourtester.path / "__pycache__") /home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.15.0/tests/test_pytest_randomly.py:549: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib64/python3.9/shutil.py:724: in rmtree onerror(os.lstat, path, sys.exc_info()) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ path = PosixPath('/tmp/pytest-of-tkloczko/pytest-212/test_it_runs_before_stepwise0/__pycache__') ignore_errors = False onerror = .onerror at 0x7ff0ff854310> def rmtree(path, ignore_errors=False, onerror=None): """Recursively delete a directory tree. If ignore_errors is set, errors are ignored; otherwise, if onerror is set, it is called to handle the error with arguments (func, path, exc_info) where func is platform and implementation dependent; path is the argument to that function that caused it to fail; and exc_info is a tuple returned by sys.exc_info(). If ignore_errors is false and onerror is None, an exception is raised. """ sys.audit("shutil.rmtree", path) if ignore_errors: def onerror(*args): pass elif onerror is None: def onerror(*args): raise if _use_fd_functions: # While the unsafe rmtree works fine on bytes, the fd based does not. if isinstance(path, bytes): path = os.fsdecode(path) # Note: To guard against symlink races, we use the standard # lstat()/open()/fstat() trick. try: > orig_st = os.lstat(path) E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-tkloczko/pytest-212/test_it_runs_before_stepwise0/__pycache__' /usr/lib64/python3.9/shutil.py:722: FileNotFoundError ----------------------------- Captured stdout call ----------------------------- ============================= test session starts ============================== platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0 -- /usr/bin/python3 cachedir: .pytest_cache Using --randomly-seed=1 rootdir: /tmp/pytest-of-tkloczko/pytest-212/test_it_runs_before_stepwise0 configfile: pytest.ini plugins: pytest_randomly-3.15.0, Faker-24.2.0, forked-1.6.0, xdist-3.5.0 collecting ... collected 2 items stepwise: no previously failed tests, not skipping. test_one.py::test_b FAILED =================================== FAILURES =================================== ____________________________________ test_b ____________________________________ def test_b(): > assert 0 E assert 0 test_one.py:6: AssertionError =========================== short test summary info ============================ FAILED test_one.py::test_b - assert 0 !!!!!!!! Interrupted: Test failed, continuing from this test next run. !!!!!!!!! ============================== 1 failed in 0.13s =============================== ______________________________ test_model_bakery _______________________________ ourtester = def test_model_bakery(ourtester): """ Rather than set up models, just check the random generator it uses is set between two tests to output the same number. """ ourtester.makepyfile( test_one=""" from model_bakery.random_gen import baker_random def test_a(): test_a.num = baker_random.random() if hasattr(test_b, 'num'): assert test_a.num == test_b.num def test_b(): test_b.num = baker_random.random() if hasattr(test_a, 'num'): assert test_b.num == test_a.num """ ) out = ourtester.runpytest("--randomly-seed=1") > out.assert_outcomes(passed=2) E AssertionError: assert {'errors': 1,...pped': 0, ...} == {'errors': 0,...pped': 0, ...} E E Omitting 4 identical items, use -vv to show E Differing items: E {'errors': 1} != {'errors': 0} E {'passed': 0} != {'passed': 2} E Use -v to get more diff /home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.15.0/tests/test_pytest_randomly.py:696: AssertionError ----------------------------- Captured stdout call ----------------------------- ============================= test session starts ============================== platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0 Using --randomly-seed=1 rootdir: /tmp/pytest-of-tkloczko/pytest-212/test_model_bakery0 configfile: pytest.ini plugins: pytest_randomly-3.15.0, Faker-24.2.0, forked-1.6.0, xdist-3.5.0 collected 0 items / 1 error ==================================== ERRORS ==================================== _________________________ ERROR collecting test_one.py _________________________ ImportError while importing test module '/tmp/pytest-of-tkloczko/pytest-212/test_model_bakery0/test_one.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib64/python3.9/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) test_one.py:1: in from model_bakery.random_gen import baker_random E ModuleNotFoundError: No module named 'model_bakery' =========================== short test summary info ============================ ERROR test_one.py !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! =============================== 1 error in 0.08s =============================== =========================== short test summary info ============================ FAILED tests/test_pytest_randomly.py::test_entrypoint_injection - assert [] =... FAILED tests/test_pytest_randomly.py::test_it_runs_before_stepwise - FileNotF... FAILED tests/test_pytest_randomly.py::test_model_bakery - AssertionError: ass... ======================== 3 failed, 34 passed in 13.08s ========================= ```

List of installed modules in build env:

```console Package Version ------------------ ----------- build 1.1.1 distro 1.9.0 dnf 4.19.0 exceptiongroup 1.1.3 execnet 2.0.2 factory_boy 3.3.0 Faker 24.2.0 gpg 1.23.2 importlib_metadata 7.0.1 iniconfig 2.0.0 installer 0.7.0 libdnf 0.73.0 numpy 1.26.5 packaging 24.0 pluggy 1.4.0 py 1.11.0 pyproject_hooks 1.0.0 pytest 8.1.1 pytest-forked 1.6.0 pytest-xdist 3.5.0 python-dateutil 2.9.0.post0 setuptools 69.1.1 tomli 2.0.1 wheel 0.43.0 zipp 3.17.0 ```

Please let me know if you need more details or want me to perform some diagnostics.

[adamchainz]

I only support running the tests with the tox setup in the repo. Any other attempt to run the tests is not guaranteed.

[kloczek]

OK will add note in my spec to not bother you anymore about pytest issues. Is it any particular reason what it fails in case of not using tox? 🤔 I'm asking because on scale +12k already packaged modules I cannot find to much examples of pytest vs. tox (generally 99% works with pytest) so still cannot recognise possible causes of such differences😞

PS I'm trying to use pytest in as many packaged modules as I can because I can preriom additional test by altering pytest params and list of additional pytest extensions and tox perform all tests in static set of modules.

[kloczek]

Just tested using tox and look like it fails for me as well (other way but fails)

Here is tox output:

```console + /usr/bin/pytest -ra -m 'not network' ==================================================================================== test session starts ==================================================================================== platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0 Using --randomly-seed=1336145680 rootdir: /home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.15.0 configfile: pyproject.toml plugins: pytest_randomly-3.15.0, Faker-24.2.0, forked-1.6.0, xdist-3.5.0 collected 37 items tests/test_pytest_randomly.py .......F............FF............... [100%] ========================================================================================= FAILURES ========================================================================================== _________________________________________________________________________________ test_entrypoint_injection _________________________________________________________________________________ pytester = , monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fb8dd360880> def test_entrypoint_injection(pytester, monkeypatch): """Test that registered entry points are seeded""" (pytester.path / "test_one.py").write_text("def test_one(): pass\n") class _FakeEntryPoint: """Minimal surface of Entry point API to allow testing""" def __init__(self, name: str, obj: mock.Mock) -> None: self.name = name self._obj = obj def load(self) -> mock.Mock: return self._obj entry_points: list[_FakeEntryPoint] = [] def fake_entry_points(*, group): return entry_points monkeypatch.setattr(pytest_randomly, "entry_points", fake_entry_points) reseed = mock.Mock() entry_points.append(_FakeEntryPoint("test_seeder", reseed)) # Need to run in-process so that monkeypatching works pytester.runpytest_inprocess("--randomly-seed=1") > assert reseed.mock_calls == [ mock.call(1), mock.call(1), mock.call(0), mock.call(1), mock.call(2), ] E assert [] == [call(1), cal...l(1), call(2)] E E Right contains 5 more items, first extra item: call(1) E Use -v to get more diff /home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.15.0/tests/test_pytest_randomly.py:769: AssertionError ----------------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------------ ============================= test session starts ============================== platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0 Using --randomly-seed=1 rootdir: /tmp/pytest-of-tkloczko/pytest-118/test_entrypoint_injection0 plugins: pytest_randomly-3.15.0, Faker-24.2.0, forked-1.6.0, xdist-3.5.0 collected 1 item test_one.py . [100%] ============================== 1 passed in 0.03s =============================== _______________________________________________________________________________ test_it_runs_before_stepwise ________________________________________________________________________________ ourtester = def test_it_runs_before_stepwise(ourtester): ourtester.makepyfile( test_one=""" def test_a(): assert 0 def test_b(): assert 0 """ ) out = ourtester.runpytest("-v", "--randomly-seed=1", "--stepwise") out.assert_outcomes(failed=1) # Now make test_b pass ourtester.makepyfile( test_one=""" def test_a(): assert 0 def test_b(): assert 1 """ ) > shutil.rmtree(ourtester.path / "__pycache__") /home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.15.0/tests/test_pytest_randomly.py:549: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib64/python3.9/shutil.py:724: in rmtree onerror(os.lstat, path, sys.exc_info()) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ path = PosixPath('/tmp/pytest-of-tkloczko/pytest-118/test_it_runs_before_stepwise0/__pycache__'), ignore_errors = False, onerror = .onerror at 0x7fb8dc55f790> def rmtree(path, ignore_errors=False, onerror=None): """Recursively delete a directory tree. If ignore_errors is set, errors are ignored; otherwise, if onerror is set, it is called to handle the error with arguments (func, path, exc_info) where func is platform and implementation dependent; path is the argument to that function that caused it to fail; and exc_info is a tuple returned by sys.exc_info(). If ignore_errors is false and onerror is None, an exception is raised. """ sys.audit("shutil.rmtree", path) if ignore_errors: def onerror(*args): pass elif onerror is None: def onerror(*args): raise if _use_fd_functions: # While the unsafe rmtree works fine on bytes, the fd based does not. if isinstance(path, bytes): path = os.fsdecode(path) # Note: To guard against symlink races, we use the standard # lstat()/open()/fstat() trick. try: > orig_st = os.lstat(path) E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-tkloczko/pytest-118/test_it_runs_before_stepwise0/__pycache__' /usr/lib64/python3.9/shutil.py:722: FileNotFoundError ----------------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------------ ============================= test session starts ============================== platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0 -- /usr/bin/python3 cachedir: .pytest_cache Using --randomly-seed=1 rootdir: /tmp/pytest-of-tkloczko/pytest-118/test_it_runs_before_stepwise0 configfile: pytest.ini plugins: pytest_randomly-3.15.0, Faker-24.2.0, forked-1.6.0, xdist-3.5.0 collecting ... collected 2 items stepwise: no previously failed tests, not skipping. test_one.py::test_b FAILED =================================== FAILURES =================================== ____________________________________ test_b ____________________________________ def test_b(): > assert 0 E assert 0 test_one.py:6: AssertionError =========================== short test summary info ============================ FAILED test_one.py::test_b - assert 0 !!!!!!!! Interrupted: Test failed, continuing from this test next run. !!!!!!!!! ============================== 1 failed in 0.15s =============================== _____________________________________________________________________________________ test_model_bakery _____________________________________________________________________________________ ourtester = def test_model_bakery(ourtester): """ Rather than set up models, just check the random generator it uses is set between two tests to output the same number. """ ourtester.makepyfile( test_one=""" from model_bakery.random_gen import baker_random def test_a(): test_a.num = baker_random.random() if hasattr(test_b, 'num'): assert test_a.num == test_b.num def test_b(): test_b.num = baker_random.random() if hasattr(test_a, 'num'): assert test_b.num == test_a.num """ ) out = ourtester.runpytest("--randomly-seed=1") > out.assert_outcomes(passed=2) E AssertionError: assert {'errors': 1,...pped': 0, ...} == {'errors': 0,...pped': 0, ...} E E Omitting 4 identical items, use -vv to show E Differing items: E {'passed': 0} != {'passed': 2} E {'errors': 1} != {'errors': 0} E Use -v to get more diff /home/tkloczko/rpmbuild/BUILD/pytest-randomly-3.15.0/tests/test_pytest_randomly.py:696: AssertionError ----------------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------------ ============================= test session starts ============================== platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0 Using --randomly-seed=1 rootdir: /tmp/pytest-of-tkloczko/pytest-118/test_model_bakery0 configfile: pytest.ini plugins: pytest_randomly-3.15.0, Faker-24.2.0, forked-1.6.0, xdist-3.5.0 collected 0 items / 1 error ==================================== ERRORS ==================================== _________________________ ERROR collecting test_one.py _________________________ ImportError while importing test module '/tmp/pytest-of-tkloczko/pytest-118/test_model_bakery0/test_one.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib64/python3.9/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) test_one.py:1: in from model_bakery.random_gen import baker_random E ModuleNotFoundError: No module named 'model_bakery' =========================== short test summary info ============================ ERROR test_one.py !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! =============================== 1 error in 0.09s =============================== ================================================================================== short test summary info ================================================================================== FAILED tests/test_pytest_randomly.py::test_entrypoint_injection - assert [] == [call(1), cal...l(1), call(2)] FAILED tests/test_pytest_randomly.py::test_it_runs_before_stepwise - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-tkloczko/pytest-118/test_it_runs_before_stepwise0/__pycache__' FAILED tests/test_pytest_randomly.py::test_model_bakery - AssertionError: assert {'errors': 1,...pped': 0, ...} == {'errors': 0,...pped': 0, ...} =============================================================================== 3 failed, 34 passed in 12.82s =============================================================================== ```