search

jaraco / pip-run

pip-run - dynamic dependency loader for Python
MIT License
136 stars 19 forks source link

9.4.0: pytest is failing in two units #65

Closed kloczek closed 1 year ago

kloczek commented 1 year ago

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

Here is pytest output:

```console + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pip-run-9.4.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pip-run-9.4.0-2.fc35.x86_64/usr/lib/python3.8/site-packages + /usr/bin/pytest -ra -m 'not network' ==================================================================================== test session starts ==================================================================================== platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0 rootdir: /home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0, configfile: pytest.ini collected 37 items pip_run/commands.py ... [ 8%] pip_run/deps.py .. [ 13%] pip_run/persist.py . [ 16%] pip_run/read-deps.py . [ 18%] pip_run/scripts.py .. [ 24%] tests/test_deps.py .......... [ 51%] tests/test_launch.py ..x.. [ 64%] tests/test_scripts.py F..........F. [100%] ========================================================================================= FAILURES ========================================================================================== _____________________________________________________________________________________ test_pkg_imported _____________________________________________________________________________________ tmp_path = PosixPath('/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0') def test_pkg_imported(tmp_path): """ Create a script that loads cython and ensure it runs. """ jaraco.path.build( { 'script': DALS( """ import path print("Successfully imported path.py") """ ) }, tmp_path, ) script = tmp_path / 'script' pip_args = ['path.py'] cmd = [sys.executable, '-m', 'pip-run'] + pip_args + ['--', str(script)] > out = subprocess.check_output(cmd, text=True) tests/test_scripts.py:35: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib64/python3.8/subprocess.py:415: in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ input = None, capture_output = False, timeout = None, check = True popenargs = (['/usr/bin/python3', '-m', 'pip-run', 'path.py', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0/script'],), kwargs = {'stdout': -1, 'text': True} process = , stdout = '', stderr = None, retcode = 1 def run(*popenargs, input=None, capture_output=False, timeout=None, check=False, **kwargs): """Run command with arguments and return a CompletedProcess instance. The returned instance will have attributes args, returncode, stdout and stderr. By default, stdout and stderr are not captured, and those attributes will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them. If check is True and the exit code was non-zero, it raises a CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute, and output & stderr attributes if those streams were captured. If timeout is given, and the process takes too long, a TimeoutExpired exception will be raised. There is an optional argument "input", allowing you to pass bytes or a string to the subprocess's stdin. If you use this argument you may not also use the Popen constructor's "stdin" argument, as it will be used internally. By default, all communication is in bytes, and therefore any "input" should be bytes, and the stdout and stderr will be bytes. If in text mode, any "input" should be a string, and stdout and stderr will be strings decoded according to locale encoding, or by "encoding" if set. Text mode is triggered by setting any of text, encoding, errors or universal_newlines. The other arguments are the same as for the Popen constructor. """ if input is not None: if kwargs.get('stdin') is not None: raise ValueError('stdin and input arguments may not both be used.') kwargs['stdin'] = PIPE if capture_output: if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None: raise ValueError('stdout and stderr arguments may not be used ' 'with capture_output.') kwargs['stdout'] = PIPE kwargs['stderr'] = PIPE with Popen(*popenargs, **kwargs) as process: try: stdout, stderr = process.communicate(input, timeout=timeout) except TimeoutExpired as exc: process.kill() if _mswindows: # Windows accumulates the output in a single blocking # read() call run on child threads, with the timeout # being done in a join() on those threads. communicate() # _after_ kill() is required to collect that and add it # to the exception. exc.stdout, exc.stderr = process.communicate() else: # POSIX _communicate already populated the output so # far into the TimeoutExpired exception. process.wait() raise except: # Including KeyboardInterrupt, communicate handled that. process.kill() # We don't call process.wait() as .__exit__ does that for us. raise retcode = process.poll() if check and retcode: > raise CalledProcessError(retcode, process.args, output=stdout, stderr=stderr) E subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', 'path.py', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0/script']' returned non-zero exit status 1. /usr/lib64/python3.8/subprocess.py:516: CalledProcessError ----------------------------------------------------------------------------------- Captured stderr call ------------------------------------------------------------------------------------ WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/ WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/ WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/ WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/ WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /simple/path-py/ ERROR: Could not find a version that satisfies the requirement path.py (from versions: none) ERROR: No matching distribution found for path.py WARNING: There was an error checking the latest version of pip. Traceback (most recent call last): File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip-run.py", line 4, in __name__ == '__main__' and run() File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/__init__.py", line 18, in run with deps.load(*deps.not_installed(pip_args)) as home: File "/usr/lib64/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/deps.py", line 74, in load Install.parse(args) and empty(target) and subprocess.check_call(cmd, env=env) File "/usr/lib64/python3.8/subprocess.py", line 364, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '('/usr/bin/python3', '-m', 'pip', 'install', '-t', PosixPath('/tmp/pip-run-2owvtu_x'), 'path.py')' returned non-zero exit status 1. ___________________________________________________________________________ test_pkg_loaded_from_alternate_index ____________________________________________________________________________ tmp_path = PosixPath('/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0') def test_pkg_loaded_from_alternate_index(tmp_path): """ Create a script that loads cython from an alternate index and ensure it runs. """ jaraco.path.build( { 'script': DALS( """ __requires__ = ['path.py'] __index_url__ = 'https://devpi.net/root/pypi/+simple/' import path print("Successfully imported path.py") """ ) }, tmp_path, ) cmd = [sys.executable, '-m', 'pip-run', '-v', '--', str(tmp_path / 'script')] > out = subprocess.check_output(cmd, text=True) tests/test_scripts.py:176: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib64/python3.8/subprocess.py:415: in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ input = None, capture_output = False, timeout = None, check = True popenargs = (['/usr/bin/python3', '-m', 'pip-run', '-v', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0/script'],), kwargs = {'stdout': -1, 'text': True} process = , stdout = 'Looking in indexes: https://devpi.net/root/pypi/+simple/\n', stderr = None, retcode = 1 def run(*popenargs, input=None, capture_output=False, timeout=None, check=False, **kwargs): """Run command with arguments and return a CompletedProcess instance. The returned instance will have attributes args, returncode, stdout and stderr. By default, stdout and stderr are not captured, and those attributes will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them. If check is True and the exit code was non-zero, it raises a CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute, and output & stderr attributes if those streams were captured. If timeout is given, and the process takes too long, a TimeoutExpired exception will be raised. There is an optional argument "input", allowing you to pass bytes or a string to the subprocess's stdin. If you use this argument you may not also use the Popen constructor's "stdin" argument, as it will be used internally. By default, all communication is in bytes, and therefore any "input" should be bytes, and the stdout and stderr will be bytes. If in text mode, any "input" should be a string, and stdout and stderr will be strings decoded according to locale encoding, or by "encoding" if set. Text mode is triggered by setting any of text, encoding, errors or universal_newlines. The other arguments are the same as for the Popen constructor. """ if input is not None: if kwargs.get('stdin') is not None: raise ValueError('stdin and input arguments may not both be used.') kwargs['stdin'] = PIPE if capture_output: if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None: raise ValueError('stdout and stderr arguments may not be used ' 'with capture_output.') kwargs['stdout'] = PIPE kwargs['stderr'] = PIPE with Popen(*popenargs, **kwargs) as process: try: stdout, stderr = process.communicate(input, timeout=timeout) except TimeoutExpired as exc: process.kill() if _mswindows: # Windows accumulates the output in a single blocking # read() call run on child threads, with the timeout # being done in a join() on those threads. communicate() # _after_ kill() is required to collect that and add it # to the exception. exc.stdout, exc.stderr = process.communicate() else: # POSIX _communicate already populated the output so # far into the TimeoutExpired exception. process.wait() raise except: # Including KeyboardInterrupt, communicate handled that. process.kill() # We don't call process.wait() as .__exit__ does that for us. raise retcode = process.poll() if check and retcode: > raise CalledProcessError(retcode, process.args, output=stdout, stderr=stderr) E subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', '-v', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0/script']' returned non-zero exit status 1. /usr/lib64/python3.8/subprocess.py:516: CalledProcessError ----------------------------------------------------------------------------------- Captured stderr call ------------------------------------------------------------------------------------ WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/ WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/ WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/ WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/ WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -3] Temporary failure in name resolution')': /root/pypi/+simple/path-py/ ERROR: Could not find a version that satisfies the requirement path.py (from versions: none) ERROR: No matching distribution found for path.py WARNING: There was an error checking the latest version of pip. Traceback (most recent call last): File "/usr/lib64/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib64/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip-run.py", line 4, in __name__ == '__main__' and run() File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/__init__.py", line 18, in run with deps.load(*deps.not_installed(pip_args)) as home: File "/usr/lib64/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/tkloczko/rpmbuild/BUILD/pip-run-9.4.0/pip_run/deps.py", line 74, in load Install.parse(args) and empty(target) and subprocess.check_call(cmd, env=env) File "/usr/lib64/python3.8/subprocess.py", line 364, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '('/usr/bin/python3', '-m', 'pip', 'install', '-t', PosixPath('/tmp/pip-run-xz8c_iz3'), '-v', '--index-url', 'https://devpi.net/root/pypi/+simple/', 'path.py')' returned non-zero exit status 1. ================================================================================== short test summary info ================================================================================== XFAIL tests/test_launch.py::test_with_path_overlay - cleanup can't occur with execv; #4 FAILED tests/test_scripts.py::test_pkg_imported - subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', 'path.py', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_imported0/script']' returned non-zero exit sta... FAILED tests/test_scripts.py::test_pkg_loaded_from_alternate_index - subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip-run', '-v', '--', '/tmp/pytest-of-tkloczko/pytest-9/test_pkg_loaded_from_alternate0/script']' returned non-zero ... =================================================================== 2 failed, 34 passed, 1 xfailed in 1647.11s (0:27:27) ==================================================================== ```

Here is list of installed modules in build env

```console Package Version ----------------------------- ----------------- alabaster 0.7.12 appdirs 1.4.4 asn1crypto 1.5.1 attrs 22.2.0 autocommand 2.2.1 Babel 2.11.0 bcrypt 3.2.2 Brlapi 0.8.3 build 0.9.0 cffi 1.15.1 charset-normalizer 3.0.1 contourpy 1.0.6 cryptography 38.0.4 cssselect 1.1.0 cycler 0.11.0 distro 1.8.0 dnspython 2.2.1 docutils 0.19 exceptiongroup 1.0.0 extras 1.0.0 fastjsonschema 2.16.1 fixtures 4.0.0 fonttools 4.38.0 gpg 1.18.0-unknown idna 3.4 imagesize 1.4.1 importlib-metadata 5.1.0 importlib-resources 5.10.1 inflect 5.0.0 iniconfig 1.1.1 jaraco.classes 3.2.3 jaraco.context 4.2.0 jaraco.functools 3.5.2 jaraco.packaging 9.1.1 jaraco.path 3.4.0 jaraco.text 3.11.0 jaraco.tidelift 1.5.0 jeepney 0.8.0 Jinja2 3.1.2 jsonschema 4.17.3 jupyter_core 5.1.1 keyring 23.11.0 kiwisolver 1.4.4 libcomps 0.1.19 louis 3.24.0 lxml 4.9.1 MarkupSafe 2.1.1 matplotlib 3.6.2 more-itertools 9.0.0 nbformat 5.7.0 numpy 1.23.1 olefile 0.46 packaging 21.3 path 16.6.0 pbr 5.9.0 pep517 0.13.0 Pillow 9.3.0 pip 22.3.1 pkgutil_resolve_name 1.3.10 platformdirs 2.6.0 pluggy 1.0.0 ply 3.11 pyasn1 0.4.8 pyasn1-modules 0.2.8 pycparser 2.21 Pygments 2.13.0 PyGObject 3.42.2 pyparsing 3.0.9 pyrsistent 0.19.2 pytest 7.2.0 python-dateutil 2.8.2 pytz 2022.4 PyYAML 6.0 requests 2.28.1 requests-toolbelt 0.10.1 rpm 4.17.0 rst.linker 2.3.1 scour 0.38.2 SecretStorage 3.3.2 setuptools 65.6.3 setuptools-scm 7.0.5 six 1.16.0 snowballstemmer 2.2.0 Sphinx 5.3.0 sphinxcontrib-applehelp 1.0.2.dev20221204 sphinxcontrib-devhelp 1.0.2.dev20221204 sphinxcontrib-htmlhelp 2.0.0 sphinxcontrib-jsmath 1.0.1.dev20221204 sphinxcontrib-qthelp 1.0.3.dev20221204 sphinxcontrib-serializinghtml 1.1.5 testtools 2.5.0 tomli 2.0.1 tpm2-pkcs11-tools 1.33.7 tpm2-pytss 1.1.0 traitlets 5.8.0 typing_extensions 4.4.0 urllib3 1.26.12 wheel 0.38.4 zipp 3.11.0 ```
jaraco commented 1 year ago

It does appear as if two of the tests are dependent on access to the network to succeed. I'll need to work out a way to exercise that behavior in an isolated environment or skip those tests.

kloczek commented 1 year ago

Just tested 10.0.1 ..

+ /usr/bin/pytest -ra -m 'not network'
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/pip-run-10.0.1, configfile: pytest.ini
collected 34 items

pip_run/commands.py ...                                                                                                                                                               [  8%]
pip_run/deps.py .                                                                                                                                                                     [ 11%]
pip_run/read-deps.py .                                                                                                                                                                [ 14%]
pip_run/scripts.py ..                                                                                                                                                                 [ 20%]
pip_run/mode/persist.py .                                                                                                                                                             [ 23%]
tests/test_deps.py ..........                                                                                                                                                         [ 52%]
tests/test_launch.py ...                                                                                                                                                              [ 61%]
tests/test_scripts.py E..........E.                                                                                                                                                   [100%]

========================================================================================== ERRORS ===========================================================================================
____________________________________________________________________________ ERROR at setup of test_pkg_imported ____________________________________________________________________________
file /home/tkloczko/rpmbuild/BUILD/pip-run-10.0.1/tests/test_scripts.py, line 16
  @pytest.mark.usefixtures('needs_internet')
  def test_pkg_imported(tmp_path):
E       fixture 'needs_internet' not found
>       available fixtures: alt_cache_dir, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, monkeypatch_session, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, reqs_files, run_mode, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/home/tkloczko/rpmbuild/BUILD/pip-run-10.0.1/tests/test_scripts.py:16
__________________________________________________________________ ERROR at setup of test_pkg_loaded_from_alternate_index ___________________________________________________________________
file /home/tkloczko/rpmbuild/BUILD/pip-run-10.0.1/tests/test_scripts.py, line 157
  @pytest.mark.usefixtures('needs_internet')
  def test_pkg_loaded_from_alternate_index(tmp_path):
E       fixture 'needs_internet' not found
>       available fixtures: alt_cache_dir, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, monkeypatch_session, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, reqs_files, run_mode, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/home/tkloczko/rpmbuild/BUILD/pip-run-10.0.1/tests/test_scripts.py:157
================================================================================== short test summary info ==================================================================================
ERROR tests/test_scripts.py::test_pkg_imported
ERROR tests/test_scripts.py::test_pkg_loaded_from_alternate_index
=============================================================================== 32 passed, 2 errors in 2.85s ================================================================================

Why not use standard pytest mark settings passed in -m "not network" by add @pytest.mark.network? 🤔
This would not require additioanl module with pytest fixture ..

kloczek commented 1 year ago

s/standard pytest mark/network mark used in many other modules test suites/

PS. I have difficulty with finding which one module provides needs_internet fixture 🤔

jaraco commented 1 year ago

Why not use the network mark used in many other modules test suites?

I guess because I was unaware of such a convention. Is it documented anywhere? I found one such place where the marker was used: https://github.com/jazzband/pip-tools/pull/917.

The needs_internet fixture works automatically. Instead of merely marking the tests and leaving it to the user to filter on it, it performs a test automatically skips tests that rely on the network. I extracted it from a technique I employed 11 years ago for the pmxbot project.

PS. I have difficulty with finding which one module provides needs_internet fixture 🤔

In order to have the fixture, you need the dependency installed. To find all of the fixtures available to the pip-run package, first install the dependencies for pip-run[testing], then run pytest --fixtures, which will emit:

------------------------------------------------------- fixtures defined from jaraco.test.http -------------------------------------------------------
needs_internet -- .tox/python/lib/python3.11/site-packages/jaraco/test/http.py:20
    no docstring available

There's also a --fixtures-per-test option:

 pip-run main $ .tox/python/bin/pytest --fixtures-per-test -k test_pkg_imported
================================================================ test session starts =================================================================
platform darwin -- Python 3.11.1, pytest-7.2.0, pluggy-1.0.0
rootdir: /Users/jaraco/code/jaraco/pip-run, configfile: pytest.ini
plugins: jaraco.test-5.2.0, black-0.3.12, mypy-0.10.3, checkdocs-2.9.0, flake8-1.1.1, enabler-2.0.0, cov-4.0.0
/Users/jaraco/code/jaraco/pip-run/.tox/python/lib/python3.11/site-packages/_pytest/nodes.py:717: PytestWarning: BlackItem is an Item subclass and should not be a collector, however its bases File are collectors.
Please split the Collectors and the Item into separate node types.
Pytest Doc example: https://docs.pytest.org/en/latest/example/nonpython.html
example pull request on a plugin: https://github.com/asmeurer/pytest-flakes/pull/40/
  warnings.warn(
collected 91 items / 89 deselected / 2 selected                                                                                                      

--------------------------------------------------------- fixtures used by test_pkg_imported ---------------------------------------------------------
------------------------------------------------------------- (tests/test_scripts.py:17) -------------------------------------------------------------
alt_cache_dir -- conftest.py:42
    no docstring available
monkeypatch_session -- conftest.py:36
    no docstring available
needs_internet -- .tox/python/lib/python3.11/site-packages/jaraco/test/http.py:20
    no docstring available
tmp_path -- .tox/python/lib/python3.11/site-packages/_pytest/tmpdir.py:203
    Return a temporary directory path object which is unique to each test
    function invocation, created as a sub directory of the base temporary
    directory.
tmp_path_factory -- .tox/python/lib/python3.11/site-packages/_pytest/tmpdir.py:188
    Return a :class:`pytest.TempPathFactory` instance for the test session.

=============================================================== 89 deselected in 0.41s ===============================================================

Of course, because fixtures use dependency injection (entry points), there's no direct path to determine which dependency is needed to satisfy usage of a fixture. Fortunately, you have pretty good signal from the commit that added the dependency at the same time as it added the usage of the fixture.

kloczek commented 1 year ago

Issue wth any public network detections is that it takes time to test that and in some cases in isolated envs it may produce mileading results. Few weeks ago when I've started building all my from packages in build env which is cut off from public network I found that many modules suppoerts alredy -m "not nework" pytest switch which allows on demand specify to skip test whihi needs anyting else than access to localhost. I have alredy padckaged +1060 pytjom modules and less than 200 are affected by building in isolated env. Amongs those ~200 about half already supports network mark so I've even modyfied my own %pytest macro to:

# Additional command over which pytest and tox could be executed.
# Exemple: to run putest/tox over "xvfb-run -a" redefin %%__py_wrapper to that value
%__py_wrapper                   %{nil}
# By tefault do not run pytest units which are using network (more than localhost access)
%with_pytest_not_network        1

%pytest %{expand:\\\
        %{set_build_flags} \\\
        PATH=%{buildroot}%{_bindir}:$PATH \\\
        LD_LIBRARY_PATH=%{buildroot}%{_libdir} \\\
        PYTHONDONTWRITEBYTECODE=1 \\\
        PBR_VERSION=%(v=%{version}; echo ${v%~*}) \\\
        PDM_PEP517_SCM_VERSION=%(v=%{version}; echo ${v%~*}) \\\
        SETUPTOOLS_SCM_PRETEND_VERSION=%(v=%{version}; echo ${v%~*}) \\\
        PYTHONPATH=${PYTHONPATH:-%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}} \\\
        %__py_wrapper \\\
        /usr/bin/pytest -ra}%{?with_pytest_not_network: -m "not network"}

Because jaraco-test 5.2.0 hs been released only few hours ago none of the other modules would be using that.

kloczek commented 1 year ago

Just tested new jaraco-test 5.2.0 and it takes 2 min until test in the fixture https://github.com/jaraco/jaraco.test/blob/8e47a5fede7785fa7652cd330ac37680ab8b36c0/jaraco/test/http.py#L8-L11 fails to skip all network dependent units 🤔 IMO a bit better would be to follow that convention to use @pytest.mark.network like in many other modules.

I've already removed in my spec files use explicit -m "not network" so here is grep otput from Fedora spec files

[tkloczko@pers-jacek SPECS.fedora]$ grep 'not network' python*
python-anyio.spec:%pytest -m "not network"
python-cartopy.spec:    -m "not network"
python-contextily.spec:%{pytest} -m 'not network'
python-dask.spec:  -m 'not network'
python-fiona.spec:%{pytest} -m "not network and not wheel" -k "not debian" -ra || :
python-matplotlib.spec:             -m 'not network' \
python-mplcairo.spec:%{python3} run-mpl-test-suite.py --tolerance=50 -m 'not network' -v -n auto \
python-pip.spec:%pytest -m 'not network' -k "$(echo $pytest_k)" \
python-rasterio.spec:%{pytest} -ra -m 'not network and not wheel' \
python-xarray.spec:  -m "not network"

However in last few weeks I found that there are many more modules which are alredy supporting that pytest mark.

jaraco commented 1 year ago

it takes 2 min until test in the fixture

That's interesting. I don't have the same experience. I disabled wifi on my mac, then ran tox -- -rs and it reported the two tests were skipped. It also failed in pytest-checkdocs, so I had to run with tox -- -p no:checkdocs -rs and the tests all passed with all internet connectivity removed (in about 3 sec).

The fact that it takes 2 minutes makes me think the network isn't actually disabled, but that packets are being dropped, so the http handler has to time out to fail. I'd welcome some improvements to the detection algorithm (currently just urllib.request.urlopen) to fail faster under a partially-degraded environment.

I've opened https://github.com/jaraco/jaraco.test/issues/5 to track using network as the indicator for tests reliant on the network. My idea is that the tests should be skipped automatically, but could also be deselected with -m not network.

kloczek commented 1 year ago

I've opened jaraco/jaraco.test#5 to track using network as the indicator for tests reliant on the network. My idea is that the tests should be skipped automatically, but could also be deselected with -m not network.

If it still would be possible tu use that convention used in other modules that would be great. 👍