Have a way to control order of plugin loading

[ionelmc]

It appears that the order of pkg_resources.iter_entry_points is pretty arbitrary.

I would like pytest to order that so I can say pytest-cov must load before pytest-benchmark. Currently I cannot use pytest-cov to measure pytest-benchmark due to the ordering issue.

[RonnyPfannschmidt]

needs consideration in pluggy i think

we might want to add a own setuptools metadata writer to pluggy, to have more informative plugin specs

[ionelmc]

We could just sort the entrypoints before calling them, no?

Would there by any problem if I decide to rename my entrypoint?

[RonnyPfannschmidt]

in my oppinion odering by name or lexical order is flawed when dependencies come into play

[ionelmc]

Just for reference, my original issue was with the imports, I wanted pytest-cov to be imported before pytest-benchmark. I have found a workaround since (albeit a bit contrived), just to measure coverage ...

[RonnyPfannschmidt]

early plugin import is planned since a while, just never found a god way to implement

[The-Compiler]

I can see how this would be useful for pytest-cov, but it shouldn't block 3.0. Changing the milestone to 3.1 for now.

[dedsm]

I'm also a victim of this, I'm using pytest_env and pytest_django, my django settings read some variables from environment (that I want to set with pytest_env), but in some computers pytest_django loads before pytest_env, so tests run without the environment.

[nicoddemus]

Back to the topic: the idea is to just somehow "mark" a plugin so it tries to be loaded as early as possible? How one would mark such a plugin?

[dedsm]

I think the easiest way is making py.test load first the plugins passed with the -p command line option before the automatic loading using pkg_resources

[nicoddemus]

@dedsm thanks for the suggestion. I think it is a valid idea, but that does not cover (heh) pytest-cov though... it is a plugin which by definition should be loaded as early as possible, independently if users have passed it into the command line or not.

I'm wondering if people have something in mind already.

[ionelmc]

One idea is to introduce another alternative entrypoint name that supports ordering. The object (a module?) that a plugin exports though that entrypoint should have some sort of priority property/attribute. Old style plugins would have default priority at 100 and ordering is done in descending order (plugins with high priority loads first). How about that?

[ionelmc]

Actually scratch that, accessing the priority attribute would imply importing module and causing undue issues. Perhaps an entrypoint just for the priority number? Hmmmm

[nicoddemus]

Would we need to have priority numbers, or something like tryfirst (where we just pile every hook which is declared as tryfirst and hope that's enough)? Priority numbers would mean some coordination between plugin authors...

Not that I have a better idea, mind you. :stuck_out_tongue_winking_eye:

[The-Compiler]

The first thing which came to mind are pytest11_tryfirst and pytest11_trylast entrypoints :laughing:

As for that vs. a priority value, I've actually talked to Holger about that one for plugin hooks, and we agreed it's the much nicer (if less powerful) mechanism, because it requires no (hard to impossible) coordination between authors. As a counter-example, nose seems to use priority values, and apparently that didn't end up being a good idea :wink:

[dedsm]

I like the tryfirst trylast idea, I think that would solve most of the cases.

for the more obscure ones, how about having a way to alter the order via the config file? a simple list that the developer can alter with a placeholder for the "rest"? something like

[pytest] plugin_order = plugin_a plugin_z all

that coupled with debug information about the order of loading would be enough for the rest of the cases.

[RonnyPfannschmidt]

i propose the concept of a pluginspec

instead of having just random modules without order, each plugin should have a spec object telling its dependencies, and the order for consideration for required/optional dependencies

tryfirst/last is imho pretty much a horrific mess

[dedsm]

the problem is none of these cases are because of lack of dependencies, the django plugin should never depend on the env plugin, same as in the case of the coverage plugin

[RonnyPfannschmidt]

so what will you do if both plugins become tryfirst?

tryfirst/last do not at all solve the problem, they just bolt a non-solution on top

what would help is a way to specifiy dependencies in plugins and out of them if your pytest,ini tells the pluginmanager, oh, in my case django needs env, you specify a topology

also we need a topology anyway - for example py.tests internal plugins can never be loaded via setuptools, and tryfirst/last couldnt fix that

[dedsm]

that's why I agree on having a really simple ordering builtin, but ultimately the control should reside in the developer because as far as I can see this problem is not common and should be treated individually

[RonnyPfannschmidt]

for me the problem is very common and currently solved by not using setuptools, but relying purely on ordered pytest-plugins specification

[dedsm]

I agree on setuptools not being a solution, but how and who should decide the topology of plugins that don't have anything to do with each other? it's naiveness thinking that every plugin will specify that relationship with all the rest of the plugins, it's not maintainable

[RonnyPfannschmidt]

there is need for 2 mechanisms

a) plugin authors spelling dependencies they do know b) plugin users spelling dependencies plugin authors cant know

[The-Compiler]

So how would pytest-cov declare that it should be loaded as early as possible?

[RonnyPfannschmidt]

i propose a priority value for an "absolute order" and before/after listing for putting topology on top of that

also a pytest.ini entry for adding extra topology "hints"

[dedsm]

and how about just ditching automatic discovery and make the developer set the order of the plugins all the time? like django middlewares and several other packages do

[RonnyPfannschmidt]

that makes adding plugins to a current env very hard for example xdist would need to be auto-added

[dedsm]

then just load everything automatically and just make clear that weird things can happen and if you get into the problem, just alter the loading order in the configuration? I don't think there's an actual need to have extra things in the plugin code and/or hooks, making it optional in the configuration would be really simple to implement IMHO, but then I'm not a core contributor or anything so just my 2 cents

[EmilStenstrom]

I just want to echo @dedsm's bug report that this undefined ordering makes tests fail for our team on some computers but not others. The only difference between them is the order at which plugins are loaded. It seems totally arbitary, and since we have no way to control it we can't use some of the plugins at all.

[fletom]

+1

[RonnyPfannschmidt]

@EmilStenstrom a workaround i currently use is - have one plugin, and that declares all other plugins in correct order - setup-tools cant be relied on

[shreyashah]

@RonnyPfannschmidt : I am trying to achieve the same thing of ordering the plugins. Trying to understand your solution better of "have one plugin, and that declares all other plugins in correct order". How do we do this? I mean, where do I declare this in my master plugin?

[RonnyPfannschmidt]

have a myplugin.py and it will declare

pytest_plugins = [
 'plugin1',
 'plugin2',
 ...,
]

[robnagler]

Running into this issue with xdist/forked. Is there a way to code a plugin to force a dependency? A simple import doesn't do it.

[RonnyPfannschmidt]

@robnagler nope, and the code you wrote to force order is fundamentally broken

without more context i cant help as the codebase you work with is hard to navigate without prior knowledge

[robnagler]

@RonnyPfannschmidt You are right. The code doesn't work.

This comment https://github.com/pytest-dev/pytest/issues/935#issuecomment-281318180 suggests we could write a plugin to load other plugins. We could do that, do you have a public example?

The PyKern plugin does not have to be an entry point. It could be explicitly registered by a config. I don't want to start hacking around until I'm sure that path would work. Again, an example would be appreciated.

I'm not sure it matters, but the goal of the plugin is to set some policies:

• Every test file has the option of having a "work" and a "data" directory, e.g. foo_test.py could have foo_work and foo_data.
• foo_work needs to be cleared before the first test in the file runs. This allows you to debug previous runs easily.
• foo_work and foo_data may contain python files so ignore them during collection.
• Finally, the plugin tracks the current test file. This is useful for managing state across the test file's cases automatically.
• And, we always run --boxed/--forked for consistency. We run Python-wrapped Fortran, C, C++, codes, and they often fail in mysterious ways. Just easier to always run in separate processes.

[robnagler]

@RonnyPfannschmidt never mind. I fixed it using conftest.py. Order seems to be guaranteed that way.

[mattbennett]

I am struggling with this. The workaround suggested in https://github.com/pytest-dev/pytest/issues/935#issuecomment-291774182 does not work if you're trying to order plugins that register themselves using setuptool entry points.

My conftest.py contains:

pytest_plugins = [
    'third_party_plugin_1',
    'third_party_plugin_2',
]

After my conftest registers the plugins, I get ValueError: Plugin already registered when the setuptools entry point subsequently runs.

Using pytest==3.7.2

[virogenesis]

How about a dependency order setting in a plugin making it load after a certain plugin in case it's defined.

That would solve the problem I am having.

depends_on = [...] Making it load after the ones defined in the list.

[RonnyPfannschmidt]

@vbarbaresi thats basically the starting point of the topology i was talking about before

[nickwilliams-eventbrite]

I would just like to throw in another user story here. We have a project using pytest-cov and pytest-django. On one machine, pytest-cov loads first, and all of our __init__.pys and settings files are marked as covered, resulting in coverage of about 90%. On another machine, pytest-django loads first, and all of our __init__.pys and settings files are marked as uncovered, resulting in a coverage of only 74%. This is a HUGE difference and makes it hard to set fail-if-coverage-under thresholds and be able to rely on them.

I really do like this suggestion I saw above:

[pytest]
plugin_order =
    plugin_a
    plugin_z
    all

That seems the most elegant to me, IMO.

[nickwilliams-eventbrite]

Related (and perhaps I should report an actual bug on this one):

If I look at the help with pytest --help, it says this:

  -p name               early-load given plugin (multi-allowed). To avoid
                        loading of plugins, use the `no:` prefix, e.g.
                        `no:doctest`.

Using that and a suggestion above, hoping I could early-load the coverage plugin, I tried that out:

$ pytest -p pytest_cov
usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --cov-report --cov-branch --cov-fail-under=90 ...........

Using -p no:pytest_cov results in the same behavior. So It appears that -p name does the exact same thing as -p no:name, which is contrary to the documentation. You can't early-load plugins.

[RonnyPfannschmidt]

The plugin setuptools name and the import name for the plugin differ, one can block by setuptools name but cant early load by it

[nickwilliams-eventbrite]

The plugin setuptools name and the import name for the plugin differ, one can block by setuptools name but cant early load by it

That's incredible confusing, and also not documented. Also, if that's the case, I can't figure out what I'm supposed to use to make pytest-cov load early. Things I've tried:

# disables plugin
$ pytest -p pytest_cov
usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: --cov-report --cov-branch --cov-fail-under=90 --cov=.......

# does not recognize it
$ pytest -p pytest-cov
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 530, in import_plugin
    __import__(importspec)
ModuleNotFoundError: No module named 'pytest-cov'

# does not recognize it
$ pytest -p cov
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 530, in import_plugin
    __import__(importspec)
ModuleNotFoundError: No module named 'cov'

# finally no error, but does not load it early
$ pytest -p coverage
================= test session starts =================
platform linux -- Python 3.7.1, pytest-4.1.1, py-1.7.0, pluggy-0.8.1
Django settings: xxxx.settings.dev_local (from environment variable)
rootdir: /srv/xxxx, inifile: setup.cfg
plugins: django-3.4.6, cov-2.6.1 .......
...
xxxx/settings/__init__.py                           36     36      0      0     0.0%   1-135
xxxx/settings/dev_local.py                          33     33      6      0     0.0%   1-125
xxxx/settings/jenkins_test.py                       24     24      4      0     0.0%   1-131
xxxx/settings/production_docker.py                  23     23      9      0     0.0%   1-125
...

# gives an error about its already being loaded
$ pytest -p pytest_cov.plugin
Traceback (most recent call last):
  File "/usr/local/bin/pytest", line 11, in <module>
    sys.exit(main())
...
ValueError: Plugin already registered: pytest_cov=<module 'pytest_cov.plugin' from '/usr/local/lib/python3.7/site-packages/pytest_cov/plugin.py'>
{'140434823318496': <_pytest.config.PytestPluginManager object at 0x7fb987c20be0>, 'pytestconfig': <_pytest.config.Config object at 0x7fb98498c5c0> ......

Is -p just broken?

[RonnyPfannschmidt]

its simply not in sync with the setuptools plugin loading mechanism which registers plugins at a different name than their import name while -p works in terms of import names

so in a sense - -p is broken

[nickwilliams-eventbrite]

Want me to file a new bug about it?

[ionelmc]

Ok so I've tried patching iter_entry_points and I realized that from pytest-cov's perspective fixing the order wouldn't really help because pytest loads all the plugins first and only later it starts calling hooks. IOW I don't have a hook to initialize coverage tracer before other plugins get imported.

Well, nothing short of doing it at import time which is way too tricky (inspecting the call stack? ugh ...).

[nicoddemus]

Now that 4.4 is out, one can force plugin loading by adding to pytest.ini:

[pytest]
addopts = -p pytest_cov

Of course, it would still be nice to have a way for plugins to handle this themselves somehow.

[Colin-b]

Now that 4.4 is out, one can force plugin loading by adding to pytest.ini:

[pytest]
addopts = -p pytest_cov

Of course, it would still be nice to have a way for plugins to handle this themselves somehow.

Hi @nicoddemus

I tried that but it does not solve the issue.

The command launched by travis is "pytest --cov", the configuration loaded is the exact one you advise.

The "pytest11" entrypoint is the root of the module leading to no coverage at all.

What's confusing here is that between your comment and the documentation, the content of the file is not the same. Also this page is a bit unclear, what specific action is suppose to start pytest-cov engine manually?

Do you have any explanation, documentation, tips to share as to what are the exact options to use in order to make sure that pytest-cov can be use to check coverage of a pytest plugin (containing a "pytest11" entrypoint) ?

Thanks again

[gaborbernat]

I've solved this by using coverage natively instead through pytest-cov, see https://github.com/pytest-dev/pytest-print/blob/eb776107d83a94e90edeb7381d679251ae45bcf0/tox.ini#L27-L33 👍

[tmax22]

2023 and still no way to do so? i'm trying to write plugin to a pytest-plugin. and the only way to do so is to make sure that my plugin is loaded before the other plugin