Selectively enable subset of plugins

pytest-dev / pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing

https://pytest.org

MIT License

11.97k stars 2.66k forks source link

Selectively enable subset of plugins #5781

Closed aklajnert closed 3 months ago

aklajnert commented 5 years ago

Currently, it is possible to disable a plugin by using -p no:plugin. There is however one feature that is missing for us, which is kind of the opposite. I want to disable all plugins but few selected.

E.g. if I'll run pytest -p only:plugin1 -p only:plugin2, the pytest should disable all the plugins except plugin1 and plugin2.

Our use case is that we have few hosts behind a firewall which makes modules installation very hard. To circumvent it, we have a common directory with pytest and some plugins. In some cases, we want to disable some of them, but since they can be added any time it would be useful to select only the plugins we need and ignore everything else.

asottile commented 5 years ago

Another idea would be to use virtualenvs:

Download the wheels for all the packages you need behind your DMZ
Set up a virtualenv and install just the packages you need pip install --no-index --find-lines file://path/to/wheels/ pytest-cov pytest

pytest itself is implemented using plugins so disabling all of them isn't really feasible

aklajnert commented 5 years ago

Actually, I meant only the third-party plugins. Those that are listed within the plugins section on the pytest summary.
Also, for debugging purposes it is sometimes useful to disable most plugins (or all).

asottile commented 5 years ago

but you can also disable internal plugins, where to draw the line?

either way I don't think this is a good idea to implement -- just make another environment without the plugins installed imo

RonnyPfannschmidt commented 5 years ago

lets rephrase the use-case a bit

i believe @aklajnert wants to switch from auto-loading distribution based plugins to explicitly loading only a select set of them

excluding/including plugins based on the distribution name as well as disabling setuptools auto-loading are both useful features, that in combination would enable the use-case @aklajnert described

aklajnert commented 5 years ago

Right. For now, we disable most of the plugins we don't want with the -p no:plugin. It doesn't work in the same way and is sometimes hard since the plugin disabling can be tricky (#5661).

drew-parsons commented 3 years ago

I would argue that the default situation should be to only load plugins (3rd-party in particular) relevant to the module being tested. At the moment pytest loads every single plugin that happens to be installed on the system, and this causes problems when there are collisions between plugins.

The biggest problem that regularly pops up is the Xvfb plugin (pytest-xvfb). It's very useful for packages that are expecting to use it, but for other packages it just generates unnecessary errors, e.g. https://github.com/qutip/qutip/issues/1639 . It would be better if pytest-xvfb were not loaded automatically.

Zac-HD commented 3 months ago

The PYTEST_DISABLE_PLUGIN_AUTOLOAD env var should solve (most of) this problem, with https://github.com/pytest-dev/pytest/issues/8969 for further configuration.