pylint-dev / pylint

It's not just a linter that annoys you!
https://pylint.readthedocs.io/en/latest/
GNU General Public License v2.0
5.26k stars 1.12k forks source link

Add randomization to test suite to find test isolation problems #9063

Open jacobtylerwalls opened 1 year ago

jacobtylerwalls commented 1 year ago

We can tackle test isolation issues in future work by adding a pytest plugin or some other solution for shuffling test cases.

Originally posted by @jacobtylerwalls in https://github.com/pylint-dev/pylint/issues/9041#issuecomment-1732605038

I'm familiar with the Django test runner's --shuffle option, but perhaps we can evaluate pytest-random-order or other plugins.

Pierre-Sassoulas commented 1 year ago

It's possible that in the case of #9041 the problem was the sys.argv content. Locally pytest does not work, vs pytest -k unbalan works, but in the CI it's python -m pytest --durations=10 --benchmark-disable --cov --cov-report= tests/ and it also did not work.