search

pytest-dev / pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
https://pytest.org
MIT License
11.64k stars 2.59k forks source link

documentation contradiction: pytest_load_initial_conftests #4482

Open JoseKilo opened 5 years ago

JoseKilo commented 5 years ago

I think there is a contradiction between this example page in the documentation and the reference of pytest_load_initial_conftests.

https://docs.pytest.org/en/latest/example/simple.html#dynamically-adding-command-line-options https://docs.pytest.org/en/latest/reference.html#_pytest.hookspec.pytest_load_initial_conftests

The examples page suggests that it can be used to " dynamically modify the command line arguments ". But the reference says This hook will not be called for conftest.py files, only for setuptools plugins.

The examples page doesn't mention anything about setuptools plugins and it gives the impression that it can be used from conftest files, like other examples from the same page.

dogonthehorizon commented 5 years ago

After struggling with the documentation linked in the OP I've found a workaround for my use case. I will have python-xdist in the requirements file for my project so shouldn't need to worry about checking if it's available.

For such a scenario I've added the following snippet to my setup.cfg file:

[tool:pytest]
# Automatically determines how many cores to use for testing
addopts = -n auto

Per the xdist documentation this will automatically determine how many cores to distribute my tests to. This produces the expected output:

> time pytest
================================================================= test session starts ==================================================================
platform linux -- Python 3.7.3, pytest-4.4.1, py-1.8.0, pluggy-0.9.0
[...]
rootdir: [...], inifile: setup.cfg
plugins: requests-mock-1.5.2, xdist-1.28.0, metadata-1.8.0, html-1.20.0, forked-1.0.2, hypothesis-4.18.0, betamax-0.8.1
gw0 [309] / gw1 [309] / gw2 [309] / gw3 [309] / gw4 [309] / gw5 [309] / gw6 [309] / gw7 [309]
RonnyPfannschmidt commented 5 years ago

It's indeed a mistake, the file comment even refers setup tools, the description however contest

dogonthehorizon commented 5 years ago

@RonnyPfannschmidt yeah, I tracked down the PRs where those comments where introduced and didn't find any additional instruction on what I needed to do as a user to enable this behavior. Rather than try and figure out how to setup a setuptools plugin I went with the above approach instead. Again, for my use case, this solves my problem completely!

marcobazzani commented 4 years ago

it's about 3 hours I'm trying to get this working without any success. how can I get pytest_load_initial_conftests to be executed should I create a plugin ? is there documentation for plugin authoring ?

seppestas commented 4 years ago

It seems like pytest doesn't allow passing options at run time, or at least I haven't find a way after almost a day of looking. Having the documentation being incorrect certainly didn't help.

This wouldn't be that much of an issue if there was an easy way to provide big configurations, but it seems the only thing that works is command line options or the ini file, which is pretty much the same since you can only use addopts.

My use-case is fairly simple: an integration test of embedded firmware using RPCs over UART. This requires quite a bit of configuration like the name and UART ports of at least 2 devices. Putting all of this in a single line in an ini file is a bit ridiculous.

mneira10 commented 3 years ago 
[tool:pytest]
# Automatically determines how many cores to use for testing
addopts = -n auto

For anyone needing to parallelize on default but also needing to override on occasion, @dogonthehorizon's comment is a great option as running pytest -n <num cores> will override whatever was specified in the addopts line