search

pytest-dev / pytest-randomly

:game_die: Pytest plugin to randomly order tests and control random.seed
MIT License
623 stars 30 forks source link

Feature enhancement to allow turning plugin on and off #550

Closed clintonsteiner closed 2 weeks ago

clintonsteiner commented 1 year ago

Description

Hello,

Using this plugin occasionally while writing new tests to prevent flakiness, but old tests still need to run without the random order. Currently accomplishing this via pytest.ini which includes -p no:randomly and a separate one you use via -c pytest-randomly.ini

This feels rather clunky, I'd like to propose creating a pytest command line argument to turn the plugin off as needed The other option is to create a bash variable PYTEST_USE_RANDOMLY which defaults to true, but can be flipped on and off as you go

I'm more than happy to make this change, but wanted to run it by the community before opening the PR.

adamchainz commented 1 year ago

to turn the plugin off as needed

This is hard to do, the plugin acts at different points in pytest's test lifecycle: collecting tests, then running modules, classes, and functions.

To force order for specific tests, you can use pytest-order, which even documents usage with pytest-randomly. Try that for your legacy tests?

clintonsteiner commented 1 year ago

Hi @adamchainz thank you for maintaining this plugin and taking the time to respond to my issue.

The project I'm using this for is massive, and adding markers to each test is not feasible. Outside of being difficult to do, do you have any objection to the ability to turn this plugin off but leave it installed via a command line argument? If would be open to the idea I will fork the repo and begin work, but didn't want to work on something that was fundamentally against the project nature.

From what I can tell, I think the change would need to occur inside https://github.com/pytest-dev/pytest-randomly/blob/main/src/pytest_randomly/__init__.py More specifically within the hook function pytest_collection_modifyitems

Let me know what your thoughts are and thanks once again.

adamchainz commented 1 year ago

I think your proposal is too niche to include in pytest-randomly right now. If others +1 this issue, maybe.

Also, you may be able to achieve what you want with a custom plugin in your project. pytest plugins are very flexible and it should be possible to make one that disables pytest-randomly by default - perhaps by telling pytest to disable the plugin, or perhaps by activating the --randomly-dont-reset-seed and --randomly-dont-reorganize options. Give that a try?

MetRonnie commented 3 months ago

First time using this plugin and my initial thoughts are: I want this to be opt-in rather than opt-out

clintonsteiner commented 2 weeks ago

Hey @adamchainz any chance you'd reconsider this? I'd be happy to do the pr if you're open to this idea. Would also be fine putting it in an optin config file somewhere if you'd prefer that

adamchainz commented 2 weeks ago

I am not really convinced that we need any new mechanism here for opting in. We already have these:

  1. Disable the plugin for some test runs with -p no:randomly.
  2. Enable the plugin only for some test runs by adding -p no:randomly in your pytest settings then adding -p randomly on the opt-in runs. (Or just install pytest-randomly for those runs.)
  3. Disable features with --randomly-dont-reset-seed or --randomly-dont-reorganize.

It doesn't seem necessary to me to introduce another mechanism and to maintain that for a long time. The plugin is only a few hundred lines long with rare changes. If you need extreme customization, you can also copy the relevant parts into your project's conftest.py.

alexandrul commented 2 weeks ago

-p no:randomly is not working in some corner cases: https://github.com/pytest-dev/pytest-randomly/issues/647