2.17.0: pytest fails in 3 units

[kloczek]

What pipdeptree version are you using?

2.17.0

Are you running pipdeptree in a virtual environment?

• [ ] Yes

Describe the problem

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix> using installer module
• run pytest with $PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>
• build is performed in env which is cut off from access to the public network (pytest is executed with -m "not network")

I'm not sure why one of those units are trying to execute /usr/bin/pipdeptree if pipdeptree is installed in $PYTHONPATH prefix.

Here is pytest output:

```console + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pipdeptree-2.17.0-2.fc37.x86_64/usr/lib64/python3.10/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pipdeptree-2.17.0-2.fc37.x86_64/usr/lib/python3.10/site-packages + /usr/bin/pytest -ra -m 'not network' ==================================================================================== test session starts ==================================================================================== platform linux -- Python 3.10.14, pytest-8.1.1, pluggy-1.4.0 rootdir: /home/tkloczko/rpmbuild/BUILD/pipdeptree-2.17.0 configfile: pyproject.toml plugins: mock-3.14.0 collected 96 items tests/_models/test_dag.py ........... [ 11%] tests/_models/test_package.py .................... [ 32%] tests/render/test_graphviz.py ... [ 35%] tests/render/test_json_tree.py .. [ 37%] tests/render/test_mermaid.py .. [ 39%] tests/render/test_render.py ..... [ 44%] tests/render/test_text.py .................... [ 65%] tests/test_cli.py ................. [ 83%] tests/test_discovery.py ... [ 86%] tests/test_non_host.py FF [ 88%] tests/test_pipdeptree.py .F [ 90%] tests/test_validate.py ......... [100%] ========================================================================================= FAILURES ========================================================================================== _______________________________________________________________________________ test_custom_interpreter[True] _______________________________________________________________________________ tmp_path = PosixPath('/tmp/pytest-of-tkloczko/pytest-344/test_custom_interpreter_True_0'), monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f8a5a2ad120> capfd = <_pytest.capture.CaptureFixture object at 0x7f8a5a2acf40>, args_joined = True @pytest.mark.parametrize("args_joined", [True, False]) def test_custom_interpreter( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capfd: pytest.CaptureFixture[str], args_joined: bool, ) -> None: result = virtualenv.cli_run([str(tmp_path / "venv"), "--activators", ""]) cmd = [sys.executable] monkeypatch.chdir(tmp_path) py = str(result.creator.exe.relative_to(tmp_path)) cmd += [f"--python={result.creator.exe}"] if args_joined else ["--python", py] monkeypatch.setattr(sys, "argv", cmd) main() out, _ = capfd.readouterr() found = {i.split("==")[0] for i in out.splitlines()} implementation = python_implementation() if implementation == "CPython": expected = {"pip", "setuptools", "wheel"} elif implementation == "PyPy": expected = {"cffi", "greenlet", "pip", "readline", "setuptools", "wheel"} # pragma: no cover else: raise ValueError(implementation) if sys.version_info >= (3, 12): expected -= {"setuptools", "wheel"} > assert found == expected, out E AssertionError: pipdeptree==2.17.0 E ├── packaging [required: >=23.1, installed: 24.0] E └── pip [required: >=23.1.2, installed: 24.0] E setuptools==69.1.1 E wheel==0.43.0 E E assert {'pipdeptree'...alled: 24.0]'} == {'pip', 'setuptools', 'wheel'} E E Extra items in the left set: E 'pipdeptree' E '├── packaging [required: >=23.1, installed: 24.0]' E '└── pip [required: >=23.1.2, installed: 24.0]' E Extra items in the right set: E 'pip' E Use -v to get more diff /home/tkloczko/rpmbuild/BUILD/pipdeptree-2.17.0/tests/test_non_host.py:41: AssertionError ______________________________________________________________________________ test_custom_interpreter[False] _______________________________________________________________________________ tmp_path = PosixPath('/tmp/pytest-of-tkloczko/pytest-344/test_custom_interpreter_False_0'), monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f8a5a2483a0> capfd = <_pytest.capture.CaptureFixture object at 0x7f8a5a1e6680>, args_joined = False @pytest.mark.parametrize("args_joined", [True, False]) def test_custom_interpreter( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capfd: pytest.CaptureFixture[str], args_joined: bool, ) -> None: result = virtualenv.cli_run([str(tmp_path / "venv"), "--activators", ""]) cmd = [sys.executable] monkeypatch.chdir(tmp_path) py = str(result.creator.exe.relative_to(tmp_path)) cmd += [f"--python={result.creator.exe}"] if args_joined else ["--python", py] monkeypatch.setattr(sys, "argv", cmd) main() out, _ = capfd.readouterr() found = {i.split("==")[0] for i in out.splitlines()} implementation = python_implementation() if implementation == "CPython": expected = {"pip", "setuptools", "wheel"} elif implementation == "PyPy": expected = {"cffi", "greenlet", "pip", "readline", "setuptools", "wheel"} # pragma: no cover else: raise ValueError(implementation) if sys.version_info >= (3, 12): expected -= {"setuptools", "wheel"} > assert found == expected, out E AssertionError: pipdeptree==2.17.0 E ├── packaging [required: >=23.1, installed: 24.0] E └── pip [required: >=23.1.2, installed: 24.0] E setuptools==69.1.1 E wheel==0.43.0 E E assert {'pipdeptree'...alled: 24.0]'} == {'pip', 'setuptools', 'wheel'} E E Extra items in the left set: E 'pipdeptree' E '├── packaging [required: >=23.1, installed: 24.0]' E '└── pip [required: >=23.1.2, installed: 24.0]' E Extra items in the right set: E 'pip' E Use -v to get more diff /home/tkloczko/rpmbuild/BUILD/pipdeptree-2.17.0/tests/test_non_host.py:41: AssertionError _______________________________________________________________________________________ test_console ________________________________________________________________________________________ def test_console() -> None: > check_call([Path(sys.executable).parent / "pipdeptree", "--help"]) tests/test_pipdeptree.py:13: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib64/python3.10/subprocess.py:364: in check_call retcode = call(*popenargs, **kwargs) /usr/lib64/python3.10/subprocess.py:345: in call with Popen(*popenargs, **kwargs) as p: /usr/lib64/python3.10/subprocess.py:971: in __init__ self._execute_child(args, executable, preexec_fn, close_fds, _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = , args = [PosixPath('/usr/bin/pipdeptree'), '--help'], executable = b'/usr/bin/pipdeptree' preexec_fn = None, close_fds = True, pass_fds = (), cwd = None, env = None, startupinfo = None, creationflags = 0, shell = False, p2cread = -1, p2cwrite = -1, c2pread = -1, c2pwrite = -1 errread = -1, errwrite = -1, restore_signals = True, gid = None, gids = None, uid = None, umask = -1, start_new_session = False def _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, restore_signals, gid, gids, uid, umask, start_new_session): """Execute program (POSIX version)""" if isinstance(args, (str, bytes)): args = [args] elif isinstance(args, os.PathLike): if shell: raise TypeError('path-like args is not allowed when ' 'shell is true') args = [args] else: args = list(args) if shell: # On Android the default shell is at '/system/bin/sh'. unix_shell = ('/system/bin/sh' if hasattr(sys, 'getandroidapilevel') else '/bin/sh') args = [unix_shell, "-c"] + args if executable: args[0] = executable if executable is None: executable = args[0] sys.audit("subprocess.Popen", executable, args, cwd, env) if (_USE_POSIX_SPAWN and os.path.dirname(executable) and preexec_fn is None and not close_fds and not pass_fds and cwd is None and (p2cread == -1 or p2cread > 2) and (c2pwrite == -1 or c2pwrite > 2) and (errwrite == -1 or errwrite > 2) and not start_new_session and gid is None and gids is None and uid is None and umask < 0): self._posix_spawn(args, executable, env, restore_signals, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite) return orig_executable = executable # For transferring possible exec failure from child to parent. # Data format: "exception name:hex errno:description" # Pickle is not used; it is complex and involves memory allocation. errpipe_read, errpipe_write = os.pipe() # errpipe_write must not be in the standard io 0, 1, or 2 fd range. low_fds_to_close = [] while errpipe_write < 3: low_fds_to_close.append(errpipe_write) errpipe_write = os.dup(errpipe_write) for low_fd in low_fds_to_close: os.close(low_fd) try: try: # We must avoid complex work that could involve # malloc or free in the child process to avoid # potential deadlocks, thus we do all this here. # and pass it to fork_exec() if env is not None: env_list = [] for k, v in env.items(): k = os.fsencode(k) if b'=' in k: raise ValueError("illegal environment variable name") env_list.append(k + b'=' + os.fsencode(v)) else: env_list = None # Use execv instead of execve. executable = os.fsencode(executable) if os.path.dirname(executable): executable_list = (executable,) else: # This matches the behavior of os._execvpe(). executable_list = tuple( os.path.join(os.fsencode(dir), executable) for dir in os.get_exec_path(env)) fds_to_keep = set(pass_fds) fds_to_keep.add(errpipe_write) self.pid = _posixsubprocess.fork_exec( args, executable_list, close_fds, tuple(sorted(map(int, fds_to_keep))), cwd, env_list, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, errpipe_read, errpipe_write, restore_signals, start_new_session, gid, gids, uid, umask, preexec_fn) self._child_created = True finally: # be sure the FD is closed no matter what os.close(errpipe_write) self._close_pipe_fds(p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite) # Wait for exec to fail or succeed; possibly raising an # exception (limited in size) errpipe_data = bytearray() while True: part = os.read(errpipe_read, 50000) errpipe_data += part if not part or len(errpipe_data) > 50000: break finally: # be sure the FD is closed no matter what os.close(errpipe_read) if errpipe_data: try: pid, sts = os.waitpid(self.pid, 0) if pid == self.pid: self._handle_exitstatus(sts) else: self.returncode = sys.maxsize except ChildProcessError: pass try: exception_name, hex_errno, err_msg = ( errpipe_data.split(b':', 2)) # The encoding here should match the encoding # written in by the subprocess implementations # like _posixsubprocess err_msg = err_msg.decode() except ValueError: exception_name = b'SubprocessError' hex_errno = b'0' err_msg = 'Bad exception data from child: {!r}'.format( bytes(errpipe_data)) child_exception_type = getattr( builtins, exception_name.decode('ascii'), SubprocessError) if issubclass(child_exception_type, OSError) and hex_errno: errno_num = int(hex_errno, 16) child_exec_never_called = (err_msg == "noexec") if child_exec_never_called: err_msg = "" # The error must be from chdir(cwd). err_filename = cwd else: err_filename = orig_executable if errno_num != 0: err_msg = os.strerror(errno_num) > raise child_exception_type(errno_num, err_msg, err_filename) E FileNotFoundError: [Errno 2] No such file or directory: PosixPath('/usr/bin/pipdeptree') /usr/lib64/python3.10/subprocess.py:1863: FileNotFoundError ================================================================================== short test summary info ================================================================================== FAILED tests/test_non_host.py::test_custom_interpreter[True] - AssertionError: pipdeptree==2.17.0 FAILED tests/test_non_host.py::test_custom_interpreter[False] - AssertionError: pipdeptree==2.17.0 FAILED tests/test_pipdeptree.py::test_console - FileNotFoundError: [Errno 2] No such file or directory: PosixPath('/usr/bin/pipdeptree') =============================================================================== 3 failed, 93 passed in 3.48s ================================================================================ ```

List of installed modules in build env:

```console Package Version ------------------ ----------- build 1.2.1 distlib 0.3.8 editables 0.5 exceptiongroup 1.1.3 filelock 3.13.3 graphviz 0.20.2 hatch-vcs 0.4.0 hatchling 1.21.1 importlib_metadata 7.1.0 iniconfig 2.0.0 installer 0.7.0 packaging 24.0 pathspec 0.12.1 platformdirs 4.2.0 pluggy 1.4.0 pyproject_hooks 1.0.0 pytest 8.1.1 pytest-mock 3.14.0 python-dateutil 2.9.0.post0 setuptools 69.2.0 setuptools-scm 8.0.4 tokenize_rt 5.2.0 tomli 2.0.1 trove-classifiers 2024.4.3 typing_extensions 4.10.0 virtualenv 20.25.0 wheel 0.43.0 zipp 3.18.1 ```

Please let me know if you need more details or want me to perform some diagnostics.

[xiacunshun]

Duplicate with https://github.com/tox-dev/pipdeptree/issues/343

[xiacunshun]

FileNotFoundError: [Errno 2] No such file or directory: PosixPath('/usr/bin/pipdeptree')

I think it is because test_console is testing /usr/bin/pipdeptree, but pipdeptree is not installed there.

[kloczek]

I think it is because test_console is testing /usr/bin/pipdeptree, but pipdeptree is not installed there.

Of course it is not installed 😄 Look .. on packaging process any package is not installed as typically it is done from non-root account 😋

[xiacunshun]

I think it is because test_console is testing /usr/bin/pipdeptree, but pipdeptree is not installed there.

Of course it is not installed 😄 Look .. on packaging process any package is not installed as typically it is done from non-root account 😋

Got you. Our test defaults to pipdeptree under /usr/bin, which may conflict with the packaging process. I noticed that %pytest and other macros will add the path where pipdeptree is "make installed" to PATH, so I think we may directly use this.

[xiacunshun]

There is another way, you can try using tox instead?

[gaborbernat]

We do not support running the test suite outside a virtual environment.

[kloczek]

There is another way, you can try using tox instead?

Sadly nope .. 😞 As because tox always performs test in statically defined env and on packaging and use %pytest rpm macro I can redefine that macro without touching rpm spec file to add for example --black (with installed pytest-black) or perform benchmarks withpytest-benchmarkpytest extension. As redefine%tox` rpm macro is as same easy as it is not so easy to alter tox settings to propagate install in venv additional components is not so easy. On top of that if something is missing in build env tox automatically tries to download .whl archived from pypi .. and I;m not interested to test any build modules against what prawideł pypi but my own rpm packages repository. I'll skip that prod builders as I wrote on tot of this ticket are intentionally cut off from access to public network.

Is it possible to just try what on $PATH instead from fixed /usr/bin? 🤔

[kloczek]

Looks like 2.18.1 is OK 👍