search

sopel-irc / sopel

:robot::speech_balloon: An easy-to-use and highly extensible IRC Bot framework. Formerly Willie.
https://sopel.chat
Other
951 stars 405 forks source link

test: explicitly specify `testpaths` #2546

Closed SnoopJ closed 8 months ago

SnoopJ commented 8 months ago

Description

This changeset specifies the testpaths parameter, so that pytest will look in more explicit locations during collection if the user did not specify a location (if the user does this, the testpaths setting is ignored). In other words, running a bare pytest will look specifically at the files the project cares about and nothing else.

This fixes a somewhat obscure race against the filesystem that @dgw unearthed when running the automated checks in parallel. In particular, what we observed is that when mypy and pytest overlap just right, pytest may end up looking at files in .mypy_cache/ during collection, after they have been deleted, presumably when the cache is being flushed. See error details below.

This was discovered by tox -p but it hypothetically affects the make workflow as well if the user allows for multiple jobs (with the -j parameter).

click for race failure details ``` dgw@ROGAlly:~/github/sopel-irc/sopel$ tox p py312-qa: FAIL ✖ in 46.16 seconds .pkg: /home/dgw/github/sopel-irc/sopel/contrib> python -m pip install --no-deps sopel-help .pkg: _optional_hooks .> python /home/dgw/.local/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta .pkg: get_requires_for_build_sdist .> python /home/dgw/.local/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta .pkg: build_sdist .> python /home/dgw/.local/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta py312-qa: /home/dgw/github/sopel-irc/sopel/contrib> python -m pip install --no-deps sopel-help py312-qa: install_package /home/dgw/github/sopel-irc/sopel/contrib> python -I -m pip install --force-reinstall --no-deps /home/dgw/github/sopel-irc/sopel/contrib/.tox/.tmp/package/86/sopel-8.0.0.dev0.tar.gz py312-qa: commands[0] /home/dgw/github/sopel-irc/sopel/contrib> make -C.. qa make: Entering directory '/home/dgw/github/sopel-irc/sopel' flake8 mypy --check-untyped-defs sopel Success: no issues found in 81 source files coverage run -m pytest -v . ================================================= test session starts ================================================== platform linux -- Python 3.12.0, pytest-7.4.3, pluggy-1.3.0 -- /home/dgw/github/sopel-irc/sopel/contrib/.tox/py312/bin/python cachedir: contrib/.tox/py312/.pytest_cache rootdir: /home/dgw/github/sopel-irc/sopel configfile: pyproject.toml plugins: sopel-8.0.0.dev0, vcr-1.0.2, requests-mock-1.9.3 collected 0 items / 1 error ======================================================== ERRORS ======================================================== ____________________________________________ ERROR collecting test session _____________________________________________ contrib/.tox/py312/lib/python3.12/site-packages/_pytest/runner.py:341: in from_call result: Optional[TResult] = func() contrib/.tox/py312/lib/python3.12/site-packages/_pytest/runner.py:372: in call = CallInfo.from_call(lambda: list(collector.collect()), "collect") contrib/.tox/py312/lib/python3.12/site-packages/_pytest/main.py:749: in collect for x in self._collectfile(path): contrib/.tox/py312/lib/python3.12/site-packages/_pytest/main.py:583: in _collectfile assert ( E AssertionError: PosixPath('/home/dgw/github/sopel-irc/sopel/.mypy_cache/3.10/urllib3/util/__init__.data.json.a77f639487600c09') is not a file (isdir=False, exists=False, islink=False) =============================================== short test summary info ================================================ ERROR - AssertionError: PosixPath('/home/dgw/github/sopel-irc/sopel/.mypy_cache/3.10/urllib3/util/__init__.data.json.a77f63... !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! =================================================== 1 error in 2.73s =================================================== make: Leaving directory '/home/dgw/github/sopel-irc/sopel' py312-qa: exit 2 (26.65 seconds) /home/dgw/github/sopel-irc/sopel/contrib> make -C.. qa pid=15412 py310-qa: FAIL ✖ in 48 seconds py310-qa: /home/dgw/github/sopel-irc/sopel/contrib> python -m pip install --no-deps sopel-help py310-qa: install_package /home/dgw/github/sopel-irc/sopel/contrib> python -I -m pip install --force-reinstall --no-deps /home/dgw/github/sopel-irc/sopel/contrib/.tox/.tmp/package/87/sopel-8.0.0.dev0.tar.gz py310-qa: commands[0] /home/dgw/github/sopel-irc/sopel/contrib> make -C.. qa make: Entering directory '/home/dgw/github/sopel-irc/sopel' flake8 mypy --check-untyped-defs sopel contrib/.tox/py310/lib/python3.10/site-packages/_pytest/compat.py:20: error: Skipping analyzing "py": module is installed, but missing library stubs or py.typed marker [import-untyped] contrib/.tox/py310/lib/python3.10/site-packages/_pytest/compat.py:20: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports Found 1 error in 1 file (checked 81 source files) make: Leaving directory '/home/dgw/github/sopel-irc/sopel' py310-qa: exit 2 (28.27 seconds) /home/dgw/github/sopel-irc/sopel/contrib> make -C.. qa pid=15431 py38-qa: OK ✔ in 1 minute 32.35 seconds py39-qa: OK ✔ in 1 minute 35.89 seconds py38-qa: OK (92.35=setup[19.73]+cmd[72.62] seconds) py39-qa: OK (95.89=setup[19.73]+cmd[76.16] seconds) py310-qa: FAIL code 2 (48.00=setup[19.73]+cmd[28.27] seconds) py311-qa: OK (96.70=setup[18.87]+cmd[77.83] seconds) py312-qa: FAIL code 2 (46.16=setup[19.51]+cmd[26.65] seconds) evaluation failed :( (96.79 seconds) make: *** [Makefile:15: test] Error 2 make: *** [Makefile:11: lint-type] Error 1 ```

As an aside, in a quick strace test against my local repository, I observed 62233 calls to stat() before this patch, and only 8140 afterwards. It doesn't really impact the runtime of the tests, but it's indicative of how much less work has to happen during collection.

Checklist