pypa / pyproject-hooks

A low-level library for calling build-backends in `pyproject.toml`-based project
https://pyproject-hooks.readthedocs.io/
MIT License
122 stars 49 forks source link

Unable to `pip install .` on a setup.py-based project #173

Closed arcivanov closed 1 year ago

arcivanov commented 1 year ago

I'm incorporating by reference this PIP issue: https://github.com/pypa/pip/issues/12131

(venv-test-3.12) bash-5.2$ ls -la
total 36
drwxr-xr-x   3 arcivanov arcivanov    81 Jul  5 16:58 .
drwxr-xr-x. 12 arcivanov arcivanov  4096 Jul  5 16:58 ..
-rwxr-xr-x   1 arcivanov arcivanov  8502 Jun 22 15:35 build.py
-rw-rw-r--   1 arcivanov arcivanov 11344 Mar  1  2020 LICENSE
-rw-r--r--   1 arcivanov arcivanov  3004 Jun 22 15:36 README.md
-rw-rw-r--   1 arcivanov arcivanov  1895 Aug 31  2021 setup.py
drwxr-xr-x   6 arcivanov arcivanov    76 Feb  1  2020 src
(venv-test-3.12) bash-5.2$ pip -vvvvvvvvvvvvv  install .
Using pip 23.1.2 from /home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip (python 3.12)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-build-tracker-a_o8dsgw
Initialized build tracking at /tmp/pip-build-tracker-a_o8dsgw
Created build tracker: /tmp/pip-build-tracker-a_o8dsgw
Entered build tracker: /tmp/pip-build-tracker-a_o8dsgw
Created temporary directory: /tmp/pip-install-ifc8yem4
Created temporary directory: /tmp/pip-ephem-wheel-cache-lnikv5l9
Processing /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
  Added file:///home/arcivanov/Documents/src/pybuilder/pybuilder/tmp to build tracker '/tmp/pip-build-tracker-a_o8dsgw'
  Created temporary directory: /tmp/pip-build-env-v01egvus
  Running command pip subprocess to install build dependencies
  Using pip 23.1.2 from /home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip (python 3.12)
  Collecting setuptools>=40.8.0
    Using cached setuptools-68.0.0-py3-none-any.whl (804 kB)
  Collecting wheel
    Using cached wheel-0.40.0-py3-none-any.whl (64 kB)
  Installing collected packages: wheel, setuptools
    Creating /tmp/pip-build-env-v01egvus/overlay/bin
    changing mode of /tmp/pip-build-env-v01egvus/overlay/bin/wheel to 755
  Successfully installed setuptools-68.0.0 wheel-0.40.0
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  PyBuilder version ${dist_version}
  Build started at 2023-07-05 16:58:53
  ------------------------------------------------------------
  [INFO]  Building pybuilder version 0.13.10.dev (0.13.10.dev20230705205853)
  [INFO]  Executing build in /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
  [INFO]  Going to execute tasks: clean, package
  [INFO]  Removing target directory /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target
  [INFO]  Processing plugin packages 'coverage>=6.0' to be installed with {'upgrade': True}
  [INFO]  Processing plugin packages 'cram' to be installed with {}
  [INFO]  Processing plugin packages 'flake8>=4.0' to be installed with {'upgrade': True}
  [INFO]  Processing plugin packages 'pdoc3>=0.8.3' to be installed with {'upgrade': True}
  [INFO]  Processing plugin packages 'pypandoc~=1.4' to be installed with {'upgrade': True}
  [INFO]  Processing plugin packages 'setuptools>=38.6.0' to be installed with {}
  [INFO]  Processing plugin packages 'sphinx' to be installed with {}
  [INFO]  Processing plugin packages 'twine>=1.15.0' to be installed with {'upgrade': True}
  [INFO]  Processing plugin packages 'unittest-xml-reporting>=3.0.4' to be installed with {'upgrade': True}
  [INFO]  Processing plugin packages 'wheel>=0.34.0' to be installed with {}
  [INFO]  Creating target 'build' VEnv in '/home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/venv/build/cpython-3.12.0.beta.3'
  [INFO]  Processing dependency packages 'pygments' to be installed with {}
  [INFO]  Creating target 'test' VEnv in '/home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/venv/test/cpython-3.12.0.beta.3'
  [INFO]  Building distribution in /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev
  [INFO]  Copying scripts to /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/scripts
  [INFO]  Copying resources matching 'LICENSE' from /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp to /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/pybuilder
  [INFO]  Writing MANIFEST.in as /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/MANIFEST.in
  [INFO]  Writing setup.py as /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/setup.py
  [INFO]  Filter resources matching pybuilder/__init__.py in /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev
  ------------------------------------------------------------
  BUILD SUCCESSFUL
  ------------------------------------------------------------
  Build Summary
               Project: pybuilder
               Version: 0.13.10.dev (0.13.10.dev20230705205853)
        Base directory: /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
          Environments:
                 Tasks: clean [0 ms] prepare [16075 ms] compile_sources [0 ms] package [79 ms]
  Build finished at 2023-07-05 16:59:09
  Build took 16 seconds (16370 ms)
  running egg_info
  creating pybuilder.egg-info
  writing pybuilder.egg-info/PKG-INFO
  writing dependency_links to pybuilder.egg-info/dependency_links.txt
  writing entry points to pybuilder.egg-info/entry_points.txt
  writing namespace_packages to pybuilder.egg-info/namespace_packages.txt
  writing top-level names to pybuilder.egg-info/top_level.txt
  writing manifest file 'pybuilder.egg-info/SOURCES.txt'
  reading manifest file 'pybuilder.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  adding license file 'LICENSE'
  writing manifest file 'pybuilder.egg-info/SOURCES.txt'
  Getting requirements to build wheel ... done
ERROR: Could not install packages due to an OSError.
Traceback (most recent call last):
  File "/home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip/_internal/commands/install.py", line 377, in run
    requirement_set = resolver.resolve(
                      ^^^^^^^^^^^^^^^^^
  File "/home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 491, in collect_root_requirements
    req = self._make_requirement_from_install_req(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 453, in _make_requirement_from_install_req
    cand = self._make_candidate_from_link(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
                                       ^^^^^^^^^^^^^^
  File "/home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 293, in __init__
    super().__init__(
  File "/home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File "/home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 304, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 516, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 631, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 69, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "/home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 48, in prepare_distribution_metadata
    self._install_build_reqs(finder)
  File "/home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 118, in _install_build_reqs
    build_reqs = self._get_build_requires_wheel()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 95, in _get_build_requires_wheel
    return backend.get_requires_for_build_wheel()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip/_internal/utils/misc.py", line 692, in get_requires_for_build_wheel
    return super().get_requires_for_build_wheel(config_settings=cs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 317, in _call_hook
    data = read_json(pjoin(td, 'output.json'))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 19, in read_json
    with open(path, encoding='utf-8') as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmptnjziu_l/output.json'
Remote version of pip: 23.1.2
Local version of pip:  23.1.2
Was pip installed by pip? True
Removed file:///home/arcivanov/Documents/src/pybuilder/pybuilder/tmp from build tracker '/tmp/pip-build-tracker-a_o8dsgw'
Removed build tracker: '/tmp/pip-build-tracker-a_o8dsgw'

Since this is a pure setup.py there is no pyproject.toml involved and legacy should keep functioning.

uranusjr commented 1 year ago

I don’t think this is a bug in pyproject-hooks, but the build backend is not working properly. But I guess it would be helpful if a better error message can be provided to point users to a likely cause.

arcivanov commented 1 year ago

Yes, that's one of the hypothesis. But importantly this same code works in Python 3.7 - 11 without a hitch so either it's related to some intended incompatible changes in 3.12 or indeed it's a failure that isn't being passed to the user obscured by absence of output.json.

pfmoore commented 1 year ago

Have you tried the suggestion I’ve made to add setuptools to build-requires? That is by far the most likely resolution at this point.

arcivanov commented 1 year ago
(venv-test-3.12) bash-5.2$ pip -vvvvvvvvvvvvv  install .
Using pip 23.1.2 from /home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip (python 3.12)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-build-tracker-a_o8dsgw
Initialized build tracking at /tmp/pip-build-tracker-a_o8dsgw
Created build tracker: /tmp/pip-build-tracker-a_o8dsgw
Entered build tracker: /tmp/pip-build-tracker-a_o8dsgw
Created temporary directory: /tmp/pip-install-ifc8yem4
Created temporary directory: /tmp/pip-ephem-wheel-cache-lnikv5l9
Processing /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
  Added file:///home/arcivanov/Documents/src/pybuilder/pybuilder/tmp to build tracker '/tmp/pip-build-tracker-a_o8dsgw'
  Created temporary directory: /tmp/pip-build-env-v01egvus
  Running command pip subprocess to install build dependencies
  Using pip 23.1.2 from /home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip (python 3.12)
  Collecting setuptools>=40.8.0
    Using cached setuptools-68.0.0-py3-none-any.whl (804 kB)
  Collecting wheel
    Using cached wheel-0.40.0-py3-none-any.whl (64 kB)
  Installing collected packages: wheel, setuptools
    Creating /tmp/pip-build-env-v01egvus/overlay/bin
    changing mode of /tmp/pip-build-env-v01egvus/overlay/bin/wheel to 755
  Successfully installed setuptools-68.0.0 wheel-0.40.0
  Installing build dependencies ... done
  Running command Getting requirements to build wheel

Both setuptools and wheel are getting installed apparently judging from the logs. Furthermore, this is a seutp.py-only build. You can't run setuptools install without setuptools. And setuptools definitely execute as you can see here:

running egg_info
  creating pybuilder.egg-info
  writing pybuilder.egg-info/PKG-INFO
  writing dependency_links to pybuilder.egg-info/dependency_links.txt
  writing entry points to pybuilder.egg-info/entry_points.txt
  writing namespace_packages to pybuilder.egg-info/namespace_packages.txt
  writing top-level names to pybuilder.egg-info/top_level.txt
  writing manifest file 'pybuilder.egg-info/SOURCES.txt'
  reading manifest file 'pybuilder.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  adding license file 'LICENSE'
  writing manifest file 'pybuilder.egg-info/SOURCES.txt'
  Getting requirements to build wheel ... done
ERROR: Could not install packages due to an OSError.
pfmoore commented 1 year ago

Looking at the code, your get_requires_for_build_wheel hook seems to be raising an unexpected exception. So there's a problem in the implementation of that hook in your backend.

It's not unreasonable to want this project to provide better reporting of errors in the build backend. Adding a catch-all exception handler to this try statement, and writing traceback information to json_out, and/or putting the write to output.json in a finally block would be a plausible fix. A PR implementing this would be welcome.

But please be clear - this is not a regression, or even a bug in any of the projects you've raised this issue against. The problem is in your build backend, and while the tools invoking your backend could provide more help to you in debugging the problem, at the end of the day the issue remains in your code.

arcivanov commented 1 year ago

We don't have get_requires_for_build_wheel hook in pyproject-less code. There is nowhere to put it in the setup.py-only build.

takluyver commented 1 year ago

setuptools implements a backend with all of these hooks, and this is used by default if there's no pyproject.toml file to specify something different. Here's the get_requires_for_build_wheel hook:

https://github.com/pypa/setuptools/blob/cd9c6a4150d87dc6a451379aea99ca65e938e451/setuptools/build_meta.py#L342-L343

Python 3.12 is the version where distutils is removed from the standard library - I wonder if this could be affecting you.

takluyver commented 1 year ago

I think the only way to get this particular error is to do something like sys.exit(0) inside the hook - this propagates as an exception in Python, so the machinery in this project doesn't write its JSON output, but it looks like a success once the process has finished, so the parent process doesn't think anything is wrong (or you'd get a different error, probably pointing more clearly to the backend).

We never really anticipated this specific possibility when writing PEP 517, because there's no obvious reason for a hook function to call sys.exit(). But setuptools works in terms of running a script, so it's has no particular need to catch SystemExit.

In principle I think the fix belongs in setuptools, because that's what's turning a script-based interface into a Python function interface. But I also don't mind fixing it in pyproject-hooks, and then if any other backend has a similar issue, we're covered.

takluyver commented 1 year ago

If my diagnosis above is correct, #176 should fix this scenario.

@arcivanov if you've got time, could you try patching that change into the copy of pyproject_hooks vendored inside your installed pip and check if that fixes things? The downside of pip's vendoring is that there isn't an easy way to just install this branch to try out.

takluyver commented 1 year ago

Meh, no, I don't think it works, because the hook needs to return something. I think the fix has to be in setuptools after all.

arcivanov commented 1 year ago

Let me put some debug printouts, or do strace.

arcivanov commented 1 year ago

So, this is a pip problem, not Python 3.12 problem. Latest pip 23.1.2 fails on both 3.12 and 3.11, but downgrading to 22.3.1 causes no problems.

arcivanov commented 1 year ago

23.0.1 also passes.

arcivanov commented 1 year ago

The failure starts occurring starting pip 23.1

arcivanov commented 1 year ago

success_python_3.11_pip_23.0.1.txt failure_python_3.11_pip_23.1.2.txt

Here are the debug statements in my setup.py:

https://github.com/pybuilder/pybuilder/pull/889/files#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7

Here are the difference in DEBUG output:

Failure (pip 23.1+):

Using pip 23.1.2 from /home/arcivanov/.pyenv/versions/3.11.4/envs/test_3.11/lib/python3.11/site-packages/pip (python 3.11)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-build-tracker-30ft0ald
Initialized build tracking at /tmp/pip-build-tracker-30ft0ald
Created build tracker: /tmp/pip-build-tracker-30ft0ald
Entered build tracker: /tmp/pip-build-tracker-30ft0ald
Created temporary directory: /tmp/pip-install-3fk36hla
Created temporary directory: /tmp/pip-ephem-wheel-cache-9cocfkm0
Processing /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
  Added file:///home/arcivanov/Documents/src/pybuilder/pybuilder/tmp to build tracker '/tmp/pip-build-tracker-30ft0ald'
  Created temporary directory: /tmp/pip-build-env-38da8sc4
  Running command pip subprocess to install build dependencies
  Using pip 23.1.2 from /home/arcivanov/.pyenv/versions/3.11.4/envs/test_3.11/lib/python3.11/site-packages/pip (python 3.11)
  Collecting setuptools>=40.8.0
    Using cached setuptools-68.0.0-py3-none-any.whl (804 kB)
  Collecting wheel
    Using cached wheel-0.40.0-py3-none-any.whl (64 kB)
  Installing collected packages: wheel, setuptools
    Creating /tmp/pip-build-env-38da8sc4/overlay/bin
    changing mode of /tmp/pip-build-env-38da8sc4/overlay/bin/wheel to 755
  Successfully installed setuptools-68.0.0 wheel-0.40.0
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  DEBUG running in /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
  PyBuilder version ${dist_version}
  Build started at 2023-07-06 15:01:57
...
  Build finished at 2023-07-06 15:02:15
  Build took 17 seconds (17999 ms)
  DEBUG moving /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/pybuilder to /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
  DEBUG moving /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/scripts to /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
  DEBUG moving /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/MANIFEST.in to /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
  DEBUG file /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/setup.py exists and will be removed
  DEBUG moving /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/setup.py to /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
  DEBUG passing to setup.py ['/home/arcivanov/.pyenv/versions/3.11.4/envs/test_3.11/bin/python3.11', 'setup.py', 'egg_info'] in /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp

Success:

Using pip 23.0.1 from /home/arcivanov/.pyenv/versions/3.11.4/envs/test_3.11/lib/python3.11/site-packages/pip (python 3.11)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-build-tracker-22j7maqk
Initialized build tracking at /tmp/pip-build-tracker-22j7maqk
Created build tracker: /tmp/pip-build-tracker-22j7maqk
Entered build tracker: /tmp/pip-build-tracker-22j7maqk
Created temporary directory: /tmp/pip-install-dftr62a1
Created temporary directory: /tmp/pip-ephem-wheel-cache-kzcjejsd
Processing /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
  Added file:///home/arcivanov/Documents/src/pybuilder/pybuilder/tmp to build tracker '/tmp/pip-build-tracker-22j7maqk'
  Running setup.py (path:/home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/setup.py) egg_info for package from file:///home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
  Created temporary directory: /tmp/pip-pip-egg-info-rp__anvx
  Running command python setup.py egg_info
  DEBUG running in /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
  PyBuilder version ${dist_version}
  Build started at 2023-07-06 15:06:33
...
  Build took 13 seconds (13440 ms)
  DEBUG moving /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/pybuilder to /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
  DEBUG moving /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/scripts to /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
  DEBUG moving /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/MANIFEST.in to /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
  DEBUG file /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/setup.py exists and will be removed
  DEBUG moving /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/setup.py to /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
  DEBUG passing to setup.py ['/home/arcivanov/.pyenv/versions/3.11.4/envs/test_3.11/bin/python3.11', 'setup.py', 'egg_info', '--egg-base', '/tmp/pip-pip-egg-info-rp__anvx'] in /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
arcivanov commented 1 year ago

As you can see in PIP >=23.1 (failure) there is:

  Running command Getting requirements to build wheel
  ...
  DEBUG passing to setup.py ['/home/arcivanov/.pyenv/versions/3.11.4/envs/test_3.11/bin/python3.11', 'setup.py', 'egg_info'] in /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp

vs

PIP <23.1 (success) there is:

  Running command python setup.py egg_info
  ...
  DEBUG passing to setup.py ['/home/arcivanov/.pyenv/versions/3.11.4/envs/test_3.11/bin/python3.11', 'setup.py', 'egg_info', '--egg-base', '/tmp/pip-pip-egg-info-rp__anvx'] in /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp

In PIP 23.1+ --egg-base is not longer passed to the setup script to indicate the location of the temporary directory where egg information is to be stored, leading to a failure.

pfmoore commented 1 year ago

So, this is a pip problem, not Python 3.12 problem

No. As you've been repeatedly told, by multiple people, it's a build backend problem. Whether that build backend is setuptools or your own code (which seems to use itself as an in-tree backend) I can't tell, and honestly I don't care much any more.

The fact that it only occurs in pip 23.1+ is almost certainly because the backend code is relying on an assumption that was never guaranteed, and has changed. Possibly to do with the fact that setuptools is no longer installed into environments by default, but that's just a guess at this point.

In PIP 23.1+ --egg-base is not longer passed to the setup script to indicate the location of the temporary directory where egg information is to be stored, leading to a failure.

This sounds like you're encountering the removal of the "legacy install path" where pip used to invoke setup.py directly, but it no longer does, instead using PEP 517 entry points to build a wheel and install that. If so, that's an intentional change, and not a bug. You will have to modify your build process to take account of this change, if it's what is hitting you.

arcivanov commented 1 year ago

This sounds like you're encountering the removal of the "legacy install path" where pip used to invoke setup.py directly, but it no longer does

Do you mean to say that nothing invokes/should invoke setup.py directly? Or that PIP-proper does not do so but some vendored dependency might?

Because setup.py egg_info is clearly invoked in pip 23.1+ as well as in pip 23.1-, just without the --egg_base. The logs clearly capture direct setup.py invocation by something from inside pip.

The setup.py in question intercepts an invocation to itself, runs PyBuilder to build the package (a setuptools-based package), copies the generated package directory along with the generated setup.py to the . used during pip install . and transparently executes the generated setup.py with the arguments passed to the original invocation of setup.py from inside pip install . : https://github.com/pybuilder/pybuilder/pull/889/files#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7L50

Finally, this issue is not for me, this is for all users that use PyBuilder-based projects without pyproject.toml with a regular (legacy) setup.py. For myself I can just disable the tests as PyBuilder does have pyproject.toml. This popped up in the E2E integration tests to ensure all functionality is preserved from one Python release to the other.

PS: I'm not trying to assign blame here, start a turf war or anything like that. That said when a dependency/vendored library fails inside PyBuilder when a user does pyb I never claim "it's not PyBuilder, it's setuptools of version x.y.z". I'm ultimately responsible for the functioning of the product as a whole, which includes whatever I decided to vendor. And I'm also 100% uncompensated for the effort.

arcivanov commented 1 year ago

PIP contains invocation of the setup.py egg_info even in main. I haven't determined if it runs or not:

https://github.com/pypa/pip/blob/main/src/pip/_internal/utils/setuptools_build.py#L134

https://github.com/pypa/pip/blob/main/src/pip/_internal/operations/build/metadata_legacy.py#L36

https://github.com/pypa/pip/blob/main/src/pip/_internal/req/req_install.py#L568

pfmoore commented 1 year ago

PIP clearly directly invokes setup.py egg_info even in main:

Yes. The transition isn't complete. But some code paths changed in 23.1.

Do you mean to say that nothing invokes/should invoke setup.py directly?

Basically yes (at least in principle). First of all, setuptools themselves have said repeatedly, and for a long time now, that direct invocation of setup.py is deprecated. We have been working on removing the direct invocations from pip for many releases now. It's a long process, because we want to give our users as much chance to migrate to a newer workflow as possible. It seems that pybuilder maybe isn't seeing the deprecation warnings, or isn't displaying them to the user, and hence this has come as a surprise to you. Sorry about that - I've no idea why that is, I don't know what pybuilder even is or does, so I'm working in the dark here.

But in terms of helping you find the issue, you still haven't provided a reproducible example of the problem. You quote pip install . but don't say what needs to be in the current directory for this to work. So all I have to go on is what you're telling me, and that has been little more than "it's not pybuilder, it's virtualenv/pip/pyproject-hooks that's at fault". So my ability to diagnose anything is limited at best.

The setup.py in question intercepts an invocation to itself, runs PyBuilder to build the package (a setuptools-based package), copies the generated package directory along with the generated setup.py to the . used during pip install . and transparently executes the generated setup.py with the arguments passed to the original invocation of setup.py from inside pip install .

Wow. That sounds incredibly complicated. I don't have anything like the time to diagnose a process like that. You will need to reduce the problem to a simple, self-contained example that demonstrates the issue without any unnecessary complexity if you want anything more than the broad generalisations I've been able to offer so far.

At an absolute minimum, just providing me with a pointer to the content of "the setup.py in question" would help. I'd probably only be able to say "that looks very unsupported, you need to fix that whether or not it's the cause of your problem", which you probably wouldn't find helpful, but at least it would give a starting point for me to understand what the heck you are doing here.

transparently executes the generated setup.py with the arguments passed to the original invocation of setup.py

That bit's not supported by setuptools (direct invocation of setup.py) or by pip (even with the legacy code path, we don't guarantee that the arguments we pass to setup.py are suitable for use in any other invocation).

I'm not trying to assign blame here, start a turf war or anything like that.

Understood. But I've been trying to offer helpful comments, even if they are suggestions that it's your code that needs to be fixed, and you've not taken any of them up, but have kept insisting that this is a "regression" in pip - frankly, without any clear evidence that this is the case (just to be 100% clear, it's not a regression just because some behaviour has changed - it could be a deliberate change as I believe this case is, or simply something that was never guaranteed to have a particular behaviour in the first place). From my perspective, it sure feels like blame 🙁

Anyway, I hope the above is of some use. If you can clarify your problem along the lines of what I've asked for above, I'll see if I can help further. But otherwise, I'm out of ideas and I'll drop the discussion at this point. Either way, best of luck working out what the problem is - it sounds like pybuilder is a pretty complicated package, and not easy to debug when things go wrong!

pradyunsg commented 1 year ago

OK, this needs a self-contained reproducer that the maintainers can test with. Is that something that you can provide?

arcivanov commented 1 year ago

Culprit was found and it is bizarre and non-trivial.

The cause of it is sys.exit(exit_code) at the end of the setup.py here even if exit_code is 0 here: https://github.com/pybuilder/pybuilder/blob/master/setup.py#L54

I'll provide a reproducible case.

Here's the difference in the outgoing trace that is installed in the pyproject_hooks/_in_process.py as follows:

def get_requires_for_build_wheel(config_settings):
    """Invoke the optional get_requires_for_build_wheel hook

    Returns [] if the hook is not defined.
    """
    backend = _build_backend()
    try:
        hook = backend.get_requires_for_build_wheel
    except AttributeError:
        return []
    else:
        print(hook, config_settings)
        import trace
        t = trace.Trace(trace=1)
        return t.runfunc(hook, config_settings)
$ diff -Nurd --color -b10 bad_exit.txt good_exit.txt 
--- bad_exit.txt        2023-07-07 00:00:43.033580092 -0400
+++ good_exit.txt       2023-07-07 00:00:55.709571023 -0400
@@ -102,23 +102,18 @@
   subprocess.py(1998):             return self.returncode
   subprocess.py(1131):         if self.returncode is None and _active is not None:
   subprocess.py(409):     if retcode:
   subprocess.py(414):     return 0
   <string>(64):  --- modulename: setup, funcname: log
   <string>(37): <string>(65): build_meta.py(495):             sys.path[:] = sys_path
   build_meta.py(496):             sys.argv[0] = sys_argv_0
   build_meta.py(322):             with Distribution.patch():
    --- modulename: contextlib, funcname: __exit__
   contextlib.py(142):         if typ is None:
-  contextlib.py(150):             if value is None:
-  contextlib.py(154):             try:
-  contextlib.py(155):                 self.gen.throw(typ, value, traceback)
+  contextlib.py(143):             try:
+  contextlib.py(144):                 next(self.gen)
    --- modulename: build_meta, funcname: patch
   build_meta.py(89):             distutils.core.Distribution = orig
-  contextlib.py(156):             except StopIteration as exc:
-  contextlib.py(161):             except RuntimeError as exc:
-  contextlib.py(179):             except BaseException as exc:
-  contextlib.py(186):                 if exc is not value:
-  contextlib.py(188):                 exc.__traceback__ = traceback
-  contextlib.py(189):                 return False
-  build_meta.py(324):         except SetupRequirementsError as e:
+  contextlib.py(145):             except StopIteration:
+  contextlib.py(146):                 return False
+  build_meta.py(327):         return requirements
   Getting requirements to build wheel ... done

I suspect that ANY setup.py that uses sys.exit will generate a problem because SystemExit exception isn't handled to generate output for output.json. But I'll need to create a reproducer for you to confirm that. Unfortunately I'm flying tomorrow so unless the information above is not enough I'll get back to this in a couple of days.

arcivanov commented 1 year ago

I think this is a pretty good confirmation:

Handling SystemExit like this solves the problem, although naturally does not return the required dependencies in case of a successful exit. I guess there has to be more handling of sys.exit somewhere around https://github.com/pypa/setuptools/blob/main/setuptools/build_meta.py#L340 as well to capture the dependencies.

def get_requires_for_build_wheel(config_settings):
    """Invoke the optional get_requires_for_build_wheel hook

    Returns [] if the hook is not defined.
    """
    backend = _build_backend()
    try:
        hook = backend.get_requires_for_build_wheel
    except AttributeError:
        return []
    else:
        print(hook, config_settings)
        import trace
        t = trace.Trace(trace=1)
        try:
            return t.runfunc(hook, config_settings)
        except SystemExit as e:
            if not e.code:
                return []
            raise

That said, one has to assume that sys.exit may be used by a setup.py even with a zero error code.

takluyver commented 1 year ago

This seems to be a minimal reproducer. Save this setup.py in a folder by itself:

from setuptools import setup
import sys

setup(
    name="pep517-test-setup-py-support",
    version="1.0",
)
sys.exit(0)

Then run pip wheel path/to/folder/ . I've just done this with pip 23.1.2, and I get the same failure as above:


Processing ./tests/samples/setup-py
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
...
  File "/home/takluyver/.local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/takluyver/.local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 317, in _call_hook
    data = read_json(pjoin(td, 'output.json'))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/takluyver/.local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 19, in read_json
    with open(path, encoding='utf-8') as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpqgj1isss/output.json'
takluyver commented 1 year ago

python -m build path/to/folder/ also fails on the example above; that gives less frontend complexity for investigating it in the backend.

arcivanov commented 1 year ago

I guess this could be solved inside setuptools though: https://github.com/pypa/setuptools/blob/main/setuptools/build_meta.py#L340.

I'll see if they'll be interested in uptaking the patch.

takluyver commented 1 year ago

I'm just writing an issue for them

takluyver commented 1 year ago

Here's the issue: https://github.com/pypa/setuptools/issues/3973

I agree that that looks like the place to fix it - feel free to have a go at a PR (but I'm not involved in setuptools, so they may want to approach it differently :shrug: )

pradyunsg commented 1 year ago

FWIW, we could also catch BaseException here.

arcivanov commented 1 year ago

Fixed in pypa/setuptools#3973

pradyunsg commented 1 year ago

For posterity, the reason you were likely seeing the behaviour change starting pip 23.1 is that pip 23.1 is when pip started using pyproject.toml-based builds in more contexts (namely: if wheel isn't installed in the global environment). This will invoke the setuptools build-backend, which itself runs setup.py egg_info under the hood -- checking if setup.py egg_info is invoked isn't going to provide much context (a different piece invokes it and, as the fix indicated, the change needed to happen on setuptools' end to not bubble up the error).