`-p no:randomly` causing tests to fail under Python 3.8 and 3.9

alexandrul commented 2 months ago

Python Version

3.9.20

pytest Version

8.3.2

Package Version

3.15.0

Description

As you can see in the following table, having `pytest-randomly` installed and using `-p no:randomly` results in failed tests but only under Python 3.8 and 3.9:		Python 3.8 / 3.9	Python 3.10 / 3.11 / 3.12
`pytest -p no:randomly tests/test_jsonreport.py::test_report_collectors`	fail	pass
`pytest tests/test_jsonreport.py::test_report_collectors`	pass	pass

To replicate the issue, checkout this repo and run the above commands in a Python 3.9 virtualenv.

alexandrul commented 2 months ago

Output for passing tests:

❯ pytest tests/test_jsonreport.py::test_report_collectors
======================================== test session starts =========================================
platform linux -- Python 3.9.20, pytest-8.3.2, pluggy-1.5.0
Using --randomly-seed=2524090775
rootdir: ~/pytest-json-report-wip-3.9/repo
plugins: metadata-3.1.1, anyio-4.4.0, randomly-3.15.0, xdist-3.6.1, flaky-3.8.1, json-report-wip-1.5.1
collected 3 items                                                                                    

tests/test_jsonreport.py ...                                                                   [100%]

========================================= 3 passed in 0.87s ==========================================

Output for failing tests:

❯ pytest -p no:randomly tests/test_jsonreport.py::test_report_collectors
======================================== test session starts =========================================
platform linux -- Python 3.9.20, pytest-8.3.2, pluggy-1.5.0
rootdir: ~/pytest-json-report-wip-3.9/repo
plugins: metadata-3.1.1, anyio-4.4.0, xdist-3.6.1, flaky-3.8.1, json-report-wip-1.5.1
collected 3 items                                                                                    

tests/test_jsonreport.py .FF                                                                   [100%]

============================================== FAILURES ==============================================
...
==================================== 2 failed, 1 passed in 0.24s =====================================

adamchainz commented 1 month ago

This sounds to me like you have some flaky tests? I don't have time to work on your project. If you can produce a minimal reproduction that proves it's an issue in pytest-randomly, sure, I can look at it.

alexandrul commented 1 month ago

I don't want you to work on my project
however, I would like to migrate to hatch and use it for tests too, instead of tox, but hatch uses your project
I couldn't find a simpler case to reproduce the issue
everything works just fine if randomly is not installed in the virtualenv
if randomly is installed and I'm not using -p no:randomly then the tests are passing under all supported Python versions
if randomly is installed and I'm using -p no:randomly then the tests are failing under Python 3.8/3.9 and are passing for Python 3.10+

I couldn't find out the root cause yet but I've seen this conditional import and it's an interesting coincidence: https://github.com/pytest-dev/pytest-randomly/blob/main/src/pytest_randomly/__init__.py#L22-L25

adamchainz commented 1 month ago

If you can provide a minimal reproducer, I can look at that. But I don't have time to clone your whole project and try debug inside unfamiliar tools like Hatch.

pytest-dev / pytest-randomly