pytest-dev / pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
https://pytest.org
MIT License
11.9k stars 2.65k forks source link

Negation filter deselects all tests if it contains a part of the path of rootdir #5889

Open quarckster opened 4 years ago

quarckster commented 4 years ago

pip list

atomicwrites (1.3.0) attrs (19.1.0) importlib-metadata (0.23) more-itertools (7.2.0) packaging (19.2) pip (9.0.1) pkg-resources (0.0.0) pluggy (0.13.0) py (1.8.0) pyparsing (2.4.2) pytest (5.2.0) setuptools (39.0.1) six (1.12.0) wcwidth (0.1.7) zipp (0.6.0)

pytest and operating system versions

pytest-5.2.0 ubuntu 18.04 x64

Reproducer

$ pwd
/tmp/another_dir
$ cat test_pytest_bug.py
def test_pytest():
    assert True
$ cd /tmp/some_dir
$ pytest -v -o testpaths="/tmp/another_dir" -k "not some_dir" 
==========================test session starts===============================
platform linux -- Python 3.6.8, pytest-5.2.0, py-1.8.0, pluggy-0.13.0 -- /tmp/test/bin/python3
cachedir: .pytest_cache
rootdir: /tmp/some_dir, testpaths: /tmp/another_dir
collected 1 item / 1 deselected    
$ cd /tmp/another_dir
$ pytest -v -o testpaths="/tmp/another_dir" -k "not some_dir"
==========================test session starts===============================
platform linux -- Python 3.6.8, pytest-5.2.0, py-1.8.0, pluggy-0.13.0 -- /tmp/test/bin/python3
cachedir: .pytest_cache
rootdir: /tmp/another_dir, testpaths: .
collected 1 item                                                                                                                                                                                                 

test_pytest_bug.py::test_pytest PASSED
RonnyPfannschmidt commented 4 years ago

https://github.com/pytest-dev/pytest/blob/8ccc0177c8be85c0725981b3a9e867baeebfbe33/src/_pytest/main.py#L363-L370 is the cause of this

rootdir being passed in as the fspath

then fscollector using https://github.com/pytest-dev/pytest/blob/8ccc0177c8be85c0725981b3a9e867baeebfbe33/src/_pytest/nodes.py#L351-L360 to determine a node name

which is followed by https://github.com/pytest-dev/pytest/blob/8ccc0177c8be85c0725981b3a9e867baeebfbe33/src/_pytest/mark/structures.py#L333-L337 turning it into a pseudo keyword marker

this is a structural bug requiring a breaking api change as the node type of session needs to change to address it

RonnyPfannschmidt commented 4 years ago

a non-root cause fix adding a pytest option/configuration to disable considering the session rootdir as a keyword can be applied more easily

gftea commented 4 years ago

the issue exist even using --rootdir parameter

some_dir $ pytest -v --rootdir="/tmp/another_dir" -k "not some_dir" ============================================================================================== test session starts =============================================================================================== platform linux -- Python 3.7.5, pytest-5.3.1, py-1.8.0, pluggy-0.13.1 -- /mnt/c/Users/ekaiqch/.pyvenv/pytest37/bin/python3.7 cachedir: .pytest_cache rootdir: /tmp/another_dir collected 0 items

@RonnyPfannschmidt, from documentation, it says below, does it means we can use anonymous as root for nodeid?

Here’s a summary what pytest uses rootdir for:

Construct nodeids during collection; each test is assigned a unique nodeid which is rooted at the rootdir and takes in account full path, class name, function name and parametrization (if any).

Is used by plugins as a stable location to store project/test run specific information; for example, the internal cache plugin creates a .pytest_cache subdirectory in rootdir to store its cross-test run state.

erheron commented 4 years ago

@RonnyPfannschmidt could You please share any thoughts about how this "non-root cause fix" might be implemented https://github.com/pytest-dev/pytest/issues/5889#issuecomment-536465517?

RonnyPfannschmidt commented 4 years ago

@erheron as per the linked code fragments prior, Session is a FSCollector with the fspath attribute set to self.rootdir

that element is also taken in as keword via that path

so the workaround would be to have session drop its keyword for its path, which is rootdir