conftest.py files in packages matching "test*" are being loaded, even if they're not in the `testpaths` list

[mattalxndr]

• [x] a detailed description of the bug or problem you are having
• [x] output of pip list from the virtual environment you are using
• [x] pytest and operating system versions
• [x] minimal example if possible

I am defining testpaths = tests in pytest.ini and that's being picked up by pytest (see pytest output below), but then it's loading a conftest.py from another directory.

Reproduction steps

Environment setup

% uname -a
Linux m 5.15.12-arch1-1 #1 SMP PREEMPT Wed, 29 Dec 2021 12:04:56 +0000 x86_64 GNU/Linux
% python --version
Python 3.10.1
% python -m venv ../venv
% ../venv/bin/pip install pytest
Collecting pytest
Using cached pytest-6.2.5-py3-none-any.whl (280 kB)
Collecting attrs>=19.2.0
Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting iniconfig
Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting packaging
Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting toml
Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting py>=1.8.2
Using cached py-1.11.0-py2.py3-none-any.whl (98 kB)
Collecting pluggy<2.0,>=0.12
Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
Using cached pyparsing-3.0.6-py3-none-any.whl (97 kB)
Installing collected packages: pyparsing, toml, py, pluggy, packaging, iniconfig, attrs, pytest
Successfully installed attrs-21.4.0 iniconfig-1.1.1 packaging-21.3 pluggy-1.0.0 py-1.11.0 pyparsing-3.0.6 pytest-6.2.5 toml-0.10.2
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '[snip]/venv/bin/python -m pip install --upgrade pip' command.
% ../venv/bin/pip list
Package    Version
---------- -------
attrs      21.4.0
iniconfig  1.1.1
packaging  21.3
pip        21.2.4
pluggy     1.0.0
py         1.11.0
pyparsing  3.0.6
pytest     6.2.5
setuptools 58.1.0
toml       0.10.2
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '[snip]/venv/bin/python -m pip install --upgrade pip' command.

The test files:

% tree
.
├── pytest.ini
├── tests
│   ├── __init__.py
│   └── test_.py
└── testsinvalid
    └── conftest.py

2 directories, 4 files

Their contents:

% find -type f | xargs -IF sh -c "echo; echo === F; cat F"

=== ./pytest.ini
[pytest]
testpaths = tests

=== ./testsinvalid/conftest.py
raise Exception('loading excluded conftest')

=== ./tests/test_.py
def test_p():
    assert True

=== ./tests/__init__.py

The command:

PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 ../venv/bin/pytest

What I expect to see

% PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 ../venv/bin/pytest
=================================================================== test session starts ====================================================================
platform linux -- Python 3.10.1, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /[snip], configfile: pytest.ini, testpaths: tests
collected 1 item

tests/test_.py .                                                                                                                                     [100%]

==================================================================== 1 passed in 0.00s =====================================================================

What I actually see

% PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 ../venv/bin/pytest
ImportError while loading conftest '[snip]/testsinvalid/conftest.py'.
testsinvalid/conftest.py:1: in <module>
raise Exception('loading excluded conftest')
E   Exception: loading excluded conftest

Of note

If I mv testsinvalid anything-without-the-prefix-tests, the problem goes away.

[The-Compiler]

I can reproduce. No idea why it's happening, though.

[mattalxndr]

@The-Compiler I narrowed it down to this block here:

https://github.com/pytest-dev/pytest/blob/1569fac603d9a50022e1474b494eebf970a2a3af/src/_pytest/config/__init__.py#L511-L513

But I don't know enough about the codebase to know what it should be doing.

[nicoddemus]

This has been added in d632a0d5c26c7c988a592e6b5a585685a576e36a, and while I agree it is surprising, it is a documented feature:

if exists, load conftest.py and test*/conftest.py relative to the directory part of the first test path.

So unless we want to somehow change this (including a deprecation period), we can close this as "worked as intended".

[The-Compiler]

That commit only moves the code around - it was initially added in 7bd60b5abb66e10cb107f79e412bfbb576e84b8b some 12 years ago:

check and load test*/conftest.py early from anchors - this makes it a bit more convenient to have command line options available from a root directory of a project that does not directly contain a conftest.py

The testpaths option was only added 5 years later, in 854e603f84dde8345cd959ad302ddb5d281e67b2. It seems reasonable to me to use that setting (if set) rather than assuming test* blindly. Still probably a backwards-incompatible change though.

[bluetech]

Reading the code, using testpaths is not right here -- the testpaths themselves are iterated and used as anchors for try_load_conftest - i.e. the idea seems to be "let's also consider test* subdirs" inside of each testpath.

[bluetech]

Hmm, sorry, I might be confusing the variable name testpaths with the config testpaths. Disregard the above :)

[dhensen]

This has been added in d632a0d, and while I agree it is surprising, it is a documented feature:

if exists, load conftest.py and test*/conftest.py relative to the directory part of the first test path.

So unless we want to somehow change this (including a deprecation period), we can close this as "worked as intended".

Is this really expected behaviour? I'm in doubt so I'll share my setup using pytest 7.0.0:

├── lambda_layer
│   ├── layer.zip
│   └── python
│       ├── numpy
│       │   └── conftest.py     <--
├── pytest.ini
├── src
│   ├── mycodehere.py
├── tests
│   └── test_my_test_code_here.py

pytest.ini looks like this:

[tool:pytest]
addopts = -m 'not e2e'
testpaths = tests

When I execute pytest it fails, because numpy's conftest wants to import dependencies that I don't have installed. (I don't want to install those dependencies, I'm making a lambda layer, so those files are maybe there or maybe not, but it feels like it should not load all conftests under my repo root when I have specified testpaths) Is this really expected behaviour?

if exists, load conftest.py and test*/conftest.py relative to the directory part of the first test path

Isn't my first test path tests as specified via testpaths in my pytest.ini? I guess when reading the manual regarding testpaths it does not really say anything about conftest loading behaviour. What to do with this?

[nicoddemus]

Is this really expected behaviour?

@dhensen this seems like a different problem, because lambda_layer doesn't match tests*.

[nicoddemus]

@dhensen, in your pytest.ini, it should be only [pytest] in the header, [tool:pytest] is only used for setup.cfg files.

[JamieMcKernanKaizen]

I'm not sure why this is a feature though because I would expect running pytest tests/ and using testpaths = tests when running pytest without any arguments to have the same behaviour, I thought that was the point of testpaths.

[nicoddemus]

I'm not sure why this is a feature though because I would expect running pytest tests/ and using testpaths = tests when running pytest without any arguments to have the same behaviour, I thought that was the point of testpaths.

I agree, seems like that feature was introduced prior to testpaths.

I guess it is possible to deprecate the behavior: we can detect a conftest.py was matched via test*/conftest.py, and issue a deprecation warning.