Open Famlam opened 3 months ago
@bluetech my first rough idea for verification would be that the culling of test-path parent directories for the Directory collection may be overzealous if one of the parent folders is path of the testdirs
it should be gracefully handled, and perhaps warn that the extra subdir entry is of no consequence we also should note that completely disabling recursion is currently a edge case we haven't worked with
We stumbled upon the same issue in Airflow - we have a way to make selective checks (based on what has changed in the incoming commit we select folders to run tests on (but we want to recursively search them).
In some cases the selective checks resulted in (roughly):
pytest tests/providers tests/providers/amazon tests/providers/google
In pytest 7.* that collected all tests/providers
tests recursively (including many other subfolders of tests/providers
- including amazon, google tests recursively as well). However with Pytest 8.3.2 that we have currently this only collects "amazon" and "google".
I downgraded pytest and checked that (as expected) the behaviour changed in Pytest 8.0.0.
I workarounded it in https://github.com/apache/airflow/pull/41499 - (I remove provider subfolders from the list) when I see I have the parent "tests/providers" also added), however, I think that behaviour is a bit unexpected - I guess it is a side effect of one of the changes in https://docs.pytest.org/en/stable/changelog.html#collection-changes, but it might be pretty disruptive if somoene had similar case - because it does not give an indication that only a subset of tests are run.
Possibly restoring old behaviour should be considered if possible.
In the project Osmose we have the following configuration:
Note especially the
testpaths = . plugins plugins/tests modules analysers
line together withnorecursedirs = *
In version 7.4.4 this meant:
In version 8.2.2 however this skips the files in the plugins folder, e.g.
The header line when running the pytest command still mentions
testpaths: ., plugins, plugins/tests, modules, analysers
I therefore think this change in behavior is likely not intended.
The directory structure, simplified, is:
Environment:
pip list
``` Package Version ------------------------- --------------- antlr4-python3-runtime 4.13.1 anyio 4.4.0 argon2-cffi 23.1.0 argon2-cffi-bindings 21.2.0 arrow 1.3.0 asttokens 2.4.1 async-lru 2.0.4 attrs 23.2.0 Babel 2.15.0 beautifulsoup4 4.12.3 bleach 6.1.0 branca 0.7.2 certifi 2024.7.4 cffi 1.16.0 charset-normalizer 3.3.2 click 8.1.7 comm 0.2.2 debugpy 1.8.2 decorator 5.1.1 defusedxml 0.7.1 exceptiongroup 1.2.2 executing 2.0.1 fastjsonschema 2.20.0 fqdn 1.5.1 future 1.0.0 geojson 3.1.0 h11 0.14.0 httpcore 1.0.5 httpx 0.27.0 idna 3.7 importlib-metadata 8.0.0 iniconfig 2.0.0 ipykernel 6.29.5 ipyleaflet 0.19.1 ipython 8.18.1 ipywidgets 8.1.3 isoduration 20.11.0 jedi 0.19.1 jinja2 3.1.4 json5 0.9.25 jsonpointer 3.0.0 jsonschema 4.23.0 jsonschema-specifications 2023.12.1 jupyter-client 8.6.2 jupyter-core 5.7.2 jupyter-events 0.10.0 jupyter-leaflet 0.19.1 jupyter-lsp 2.2.5 jupyter-server 2.14.2 jupyter-server-terminals 0.5.3 jupyterlab 4.2.3 jupyterlab-pygments 0.3.0 jupyterlab-server 2.27.2 jupyterlab-widgets 3.0.11 libarchive 0.4.8 lockfile 0.12.2 mapbox-vector-tile 1.2.1 MarkupSafe 2.1.5 matplotlib-inline 0.1.7 mccabe 0.7.0 mistune 3.0.2 mypy 1.10.1 mypy-extensions 1.0.0 nbclient 0.10.0 nbconvert 7.16.4 nbformat 5.10.4 nest-asyncio 1.6.0 nose2 0.15.1 notebook 7.2.1 notebook-shim 0.2.4 numpy 2.0.0 osmium 3.7.0 overrides 7.7.0 packaging 24.1 pandas 2.2.2 pandocfilters 1.5.1 parso 0.8.4 pexpect 4.9.0 pip 20.3.4 pkg-resources 0.0.0 platformdirs 4.2.2 pluggy 1.5.0 polib 1.2.0 prometheus-client 0.20.0 prompt-toolkit 3.0.47 protobuf 3.20.3 psutil 6.0.0 psycopg2-binary 2.9.9 ptyprocess 0.7.0 pure-eval 0.2.2 pyclipper 1.3.0.post5 pycodestyle 2.12.0 pycparser 2.22 pydocstyle 6.3.0 pyflakes 3.2.0 pygments 2.18.0 PyKOpeningHours 1.6.0 pylama 8.4.1 pyproj 3.6.1 pytest 8.2.2 python-dateutil 2.9.0.post0 python-json-logger 2.0.7 pytz 2024.1 PyYAML 6.0.1 pyzmq 26.0.3 referencing 0.35.1 regex 2024.5.15 requests 2.32.3 rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 rpds-py 0.19.0 scipy 1.13.1 Send2Trash 1.8.3 sentry-sdk 2.9.0 setuptools 44.1.1 shapely 2.0.5 six 1.16.0 sniffio 1.3.1 snowballstemmer 2.2.0 soupsieve 2.5 stack-data 0.6.3 termcolor 2.4.0 terminado 0.18.1 tiletanic 1.1.0 tinycss2 1.3.0 tomli 2.0.1 tornado 6.4.1 traitlets 5.14.3 traittypes 0.2.1 transporthours 0.0.4 types-certifi 2021.10.8.3 types-pkg-resources 0.1.3 types-python-dateutil 2.9.0.20240316 types-requests 2.32.0.20240712 types-termcolor 1.1.6.2 types-toml 0.10.8.20240310 typing-extensions 4.12.2 tzdata 2024.1 Unidecode 1.3.8 uri-template 1.3.0 urllib3 2.2.2 vt2geojson 0.2.1 wcwidth 0.2.13 webcolors 24.6.0 webencodings 0.5.1 websocket-client 1.8.0 wheel 0.34.2 widgetsnbextension 4.0.11 xmltodict 0.13.0 xyzservices 2024.6.0 zipp 3.19.2 ```Note that if I remove either
plugins
orplugins/tests
from testpaths, everything behaves as expected (except that the removed directory isn't checked of course)Related issue: https://github.com/osm-fr/osmose-backend/issues/2266