Closed arcivanov closed 1 year ago
I don’t think this is a bug in pyproject-hooks, but the build backend is not working properly. But I guess it would be helpful if a better error message can be provided to point users to a likely cause.
Yes, that's one of the hypothesis. But importantly this same code works in Python 3.7 - 11 without a hitch so either it's related to some intended incompatible changes in 3.12 or indeed it's a failure that isn't being passed to the user obscured by absence of output.json.
Have you tried the suggestion I’ve made to add setuptools to build-requires? That is by far the most likely resolution at this point.
(venv-test-3.12) bash-5.2$ pip -vvvvvvvvvvvvv install .
Using pip 23.1.2 from /home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip (python 3.12)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-build-tracker-a_o8dsgw
Initialized build tracking at /tmp/pip-build-tracker-a_o8dsgw
Created build tracker: /tmp/pip-build-tracker-a_o8dsgw
Entered build tracker: /tmp/pip-build-tracker-a_o8dsgw
Created temporary directory: /tmp/pip-install-ifc8yem4
Created temporary directory: /tmp/pip-ephem-wheel-cache-lnikv5l9
Processing /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
Added file:///home/arcivanov/Documents/src/pybuilder/pybuilder/tmp to build tracker '/tmp/pip-build-tracker-a_o8dsgw'
Created temporary directory: /tmp/pip-build-env-v01egvus
Running command pip subprocess to install build dependencies
Using pip 23.1.2 from /home/arcivanov/.pyenv/versions/3.12.0b3/envs/venv-test-3.12/lib/python3.12/site-packages/pip (python 3.12)
Collecting setuptools>=40.8.0
Using cached setuptools-68.0.0-py3-none-any.whl (804 kB)
Collecting wheel
Using cached wheel-0.40.0-py3-none-any.whl (64 kB)
Installing collected packages: wheel, setuptools
Creating /tmp/pip-build-env-v01egvus/overlay/bin
changing mode of /tmp/pip-build-env-v01egvus/overlay/bin/wheel to 755
Successfully installed setuptools-68.0.0 wheel-0.40.0
Installing build dependencies ... done
Running command Getting requirements to build wheel
Both setuptools and wheel are getting installed apparently judging from the logs. Furthermore, this is a seutp.py-only build. You can't run setuptools install without setuptools. And setuptools definitely execute as you can see here:
running egg_info
creating pybuilder.egg-info
writing pybuilder.egg-info/PKG-INFO
writing dependency_links to pybuilder.egg-info/dependency_links.txt
writing entry points to pybuilder.egg-info/entry_points.txt
writing namespace_packages to pybuilder.egg-info/namespace_packages.txt
writing top-level names to pybuilder.egg-info/top_level.txt
writing manifest file 'pybuilder.egg-info/SOURCES.txt'
reading manifest file 'pybuilder.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'pybuilder.egg-info/SOURCES.txt'
Getting requirements to build wheel ... done
ERROR: Could not install packages due to an OSError.
Looking at the code, your get_requires_for_build_wheel
hook seems to be raising an unexpected exception. So there's a problem in the implementation of that hook in your backend.
It's not unreasonable to want this project to provide better reporting of errors in the build backend. Adding a catch-all exception handler to this try
statement, and writing traceback information to json_out
, and/or putting the write to output.json
in a finally
block would be a plausible fix. A PR implementing this would be welcome.
But please be clear - this is not a regression, or even a bug in any of the projects you've raised this issue against. The problem is in your build backend, and while the tools invoking your backend could provide more help to you in debugging the problem, at the end of the day the issue remains in your code.
We don't have get_requires_for_build_wheel
hook in pyproject-less code. There is nowhere to put it in the setup.py
-only build.
setuptools implements a backend with all of these hooks, and this is used by default if there's no pyproject.toml
file to specify something different. Here's the get_requires_for_build_wheel
hook:
Python 3.12 is the version where distutils is removed from the standard library - I wonder if this could be affecting you.
I think the only way to get this particular error is to do something like sys.exit(0)
inside the hook - this propagates as an exception in Python, so the machinery in this project doesn't write its JSON output, but it looks like a success once the process has finished, so the parent process doesn't think anything is wrong (or you'd get a different error, probably pointing more clearly to the backend).
We never really anticipated this specific possibility when writing PEP 517, because there's no obvious reason for a hook function to call sys.exit()
. But setuptools works in terms of running a script, so it's has no particular need to catch SystemExit.
In principle I think the fix belongs in setuptools, because that's what's turning a script-based interface into a Python function interface. But I also don't mind fixing it in pyproject-hooks, and then if any other backend has a similar issue, we're covered.
If my diagnosis above is correct, #176 should fix this scenario.
@arcivanov if you've got time, could you try patching that change into the copy of pyproject_hooks vendored inside your installed pip and check if that fixes things? The downside of pip's vendoring is that there isn't an easy way to just install this branch to try out.
Meh, no, I don't think it works, because the hook needs to return something. I think the fix has to be in setuptools after all.
Let me put some debug printouts, or do strace.
So, this is a pip problem, not Python 3.12 problem. Latest pip 23.1.2 fails on both 3.12 and 3.11, but downgrading to 22.3.1 causes no problems.
23.0.1 also passes.
The failure starts occurring starting pip 23.1
success_python_3.11_pip_23.0.1.txt failure_python_3.11_pip_23.1.2.txt
Here are the debug statements in my setup.py
:
Here are the difference in DEBUG
output:
Failure (pip 23.1+):
Using pip 23.1.2 from /home/arcivanov/.pyenv/versions/3.11.4/envs/test_3.11/lib/python3.11/site-packages/pip (python 3.11)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-build-tracker-30ft0ald
Initialized build tracking at /tmp/pip-build-tracker-30ft0ald
Created build tracker: /tmp/pip-build-tracker-30ft0ald
Entered build tracker: /tmp/pip-build-tracker-30ft0ald
Created temporary directory: /tmp/pip-install-3fk36hla
Created temporary directory: /tmp/pip-ephem-wheel-cache-9cocfkm0
Processing /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
Added file:///home/arcivanov/Documents/src/pybuilder/pybuilder/tmp to build tracker '/tmp/pip-build-tracker-30ft0ald'
Created temporary directory: /tmp/pip-build-env-38da8sc4
Running command pip subprocess to install build dependencies
Using pip 23.1.2 from /home/arcivanov/.pyenv/versions/3.11.4/envs/test_3.11/lib/python3.11/site-packages/pip (python 3.11)
Collecting setuptools>=40.8.0
Using cached setuptools-68.0.0-py3-none-any.whl (804 kB)
Collecting wheel
Using cached wheel-0.40.0-py3-none-any.whl (64 kB)
Installing collected packages: wheel, setuptools
Creating /tmp/pip-build-env-38da8sc4/overlay/bin
changing mode of /tmp/pip-build-env-38da8sc4/overlay/bin/wheel to 755
Successfully installed setuptools-68.0.0 wheel-0.40.0
Installing build dependencies ... done
Running command Getting requirements to build wheel
DEBUG running in /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
PyBuilder version ${dist_version}
Build started at 2023-07-06 15:01:57
...
Build finished at 2023-07-06 15:02:15
Build took 17 seconds (17999 ms)
DEBUG moving /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/pybuilder to /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
DEBUG moving /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/scripts to /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
DEBUG moving /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/MANIFEST.in to /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
DEBUG file /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/setup.py exists and will be removed
DEBUG moving /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/setup.py to /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
DEBUG passing to setup.py ['/home/arcivanov/.pyenv/versions/3.11.4/envs/test_3.11/bin/python3.11', 'setup.py', 'egg_info'] in /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
Success:
Using pip 23.0.1 from /home/arcivanov/.pyenv/versions/3.11.4/envs/test_3.11/lib/python3.11/site-packages/pip (python 3.11)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-build-tracker-22j7maqk
Initialized build tracking at /tmp/pip-build-tracker-22j7maqk
Created build tracker: /tmp/pip-build-tracker-22j7maqk
Entered build tracker: /tmp/pip-build-tracker-22j7maqk
Created temporary directory: /tmp/pip-install-dftr62a1
Created temporary directory: /tmp/pip-ephem-wheel-cache-kzcjejsd
Processing /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
Added file:///home/arcivanov/Documents/src/pybuilder/pybuilder/tmp to build tracker '/tmp/pip-build-tracker-22j7maqk'
Running setup.py (path:/home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/setup.py) egg_info for package from file:///home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
Created temporary directory: /tmp/pip-pip-egg-info-rp__anvx
Running command python setup.py egg_info
DEBUG running in /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
PyBuilder version ${dist_version}
Build started at 2023-07-06 15:06:33
...
Build took 13 seconds (13440 ms)
DEBUG moving /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/pybuilder to /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
DEBUG moving /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/scripts to /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
DEBUG moving /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/MANIFEST.in to /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
DEBUG file /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/setup.py exists and will be removed
DEBUG moving /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp/target/dist/pybuilder-0.13.10.dev/setup.py to /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
DEBUG passing to setup.py ['/home/arcivanov/.pyenv/versions/3.11.4/envs/test_3.11/bin/python3.11', 'setup.py', 'egg_info', '--egg-base', '/tmp/pip-pip-egg-info-rp__anvx'] in /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
As you can see in PIP >=23.1 (failure) there is:
Running command Getting requirements to build wheel
...
DEBUG passing to setup.py ['/home/arcivanov/.pyenv/versions/3.11.4/envs/test_3.11/bin/python3.11', 'setup.py', 'egg_info'] in /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
vs
PIP <23.1 (success) there is:
Running command python setup.py egg_info
...
DEBUG passing to setup.py ['/home/arcivanov/.pyenv/versions/3.11.4/envs/test_3.11/bin/python3.11', 'setup.py', 'egg_info', '--egg-base', '/tmp/pip-pip-egg-info-rp__anvx'] in /home/arcivanov/Documents/src/pybuilder/pybuilder/tmp
In PIP 23.1+ --egg-base
is not longer passed to the setup script to indicate the location of the temporary directory where egg information is to be stored, leading to a failure.
So, this is a pip problem, not Python 3.12 problem
No. As you've been repeatedly told, by multiple people, it's a build backend problem. Whether that build backend is setuptools or your own code (which seems to use itself as an in-tree backend) I can't tell, and honestly I don't care much any more.
The fact that it only occurs in pip 23.1+ is almost certainly because the backend code is relying on an assumption that was never guaranteed, and has changed. Possibly to do with the fact that setuptools is no longer installed into environments by default, but that's just a guess at this point.
In PIP 23.1+ --egg-base is not longer passed to the setup script to indicate the location of the temporary directory where egg information is to be stored, leading to a failure.
This sounds like you're encountering the removal of the "legacy install path" where pip used to invoke setup.py
directly, but it no longer does, instead using PEP 517 entry points to build a wheel and install that. If so, that's an intentional change, and not a bug. You will have to modify your build process to take account of this change, if it's what is hitting you.
This sounds like you're encountering the removal of the "legacy install path" where pip used to invoke setup.py directly, but it no longer does
Do you mean to say that nothing invokes/should invoke setup.py
directly? Or that PIP-proper does not do so but some vendored dependency might?
Because setup.py egg_info
is clearly invoked in pip 23.1+ as well as in pip 23.1-, just without the --egg_base
. The logs clearly capture direct setup.py
invocation by something from inside pip.
The setup.py
in question intercepts an invocation to itself, runs PyBuilder to build the package (a setuptools-based package), copies the generated package directory along with the generated setup.py
to the .
used during pip install .
and transparently executes the generated setup.py
with the arguments passed to the original invocation of setup.py
from inside pip install .
: https://github.com/pybuilder/pybuilder/pull/889/files#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7L50
Finally, this issue is not for me, this is for all users that use PyBuilder-based projects without pyproject.toml
with a regular (legacy) setup.py
. For myself I can just disable the tests as PyBuilder does have pyproject.toml
. This popped up in the E2E integration tests to ensure all functionality is preserved from one Python release to the other.
PS: I'm not trying to assign blame here, start a turf war or anything like that. That said when a dependency/vendored library fails inside PyBuilder when a user does pyb
I never claim "it's not PyBuilder, it's setuptools of version x.y.z". I'm ultimately responsible for the functioning of the product as a whole, which includes whatever I decided to vendor. And I'm also 100% uncompensated for the effort.
PIP contains invocation of the setup.py egg_info
even in main. I haven't determined if it runs or not:
https://github.com/pypa/pip/blob/main/src/pip/_internal/utils/setuptools_build.py#L134
https://github.com/pypa/pip/blob/main/src/pip/_internal/operations/build/metadata_legacy.py#L36
https://github.com/pypa/pip/blob/main/src/pip/_internal/req/req_install.py#L568
PIP clearly directly invokes setup.py egg_info even in main:
Yes. The transition isn't complete. But some code paths changed in 23.1.
Do you mean to say that nothing invokes/should invoke setup.py directly?
Basically yes (at least in principle). First of all, setuptools themselves have said repeatedly, and for a long time now, that direct invocation of setup.py
is deprecated. We have been working on removing the direct invocations from pip for many releases now. It's a long process, because we want to give our users as much chance to migrate to a newer workflow as possible. It seems that pybuilder maybe isn't seeing the deprecation warnings, or isn't displaying them to the user, and hence this has come as a surprise to you. Sorry about that - I've no idea why that is, I don't know what pybuilder even is or does, so I'm working in the dark here.
But in terms of helping you find the issue, you still haven't provided a reproducible example of the problem. You quote pip install .
but don't say what needs to be in the current directory for this to work. So all I have to go on is what you're telling me, and that has been little more than "it's not pybuilder, it's virtualenv/pip/pyproject-hooks that's at fault". So my ability to diagnose anything is limited at best.
The setup.py in question intercepts an invocation to itself, runs PyBuilder to build the package (a setuptools-based package), copies the generated package directory along with the generated setup.py to the . used during pip install . and transparently executes the generated setup.py with the arguments passed to the original invocation of setup.py from inside pip install .
Wow. That sounds incredibly complicated. I don't have anything like the time to diagnose a process like that. You will need to reduce the problem to a simple, self-contained example that demonstrates the issue without any unnecessary complexity if you want anything more than the broad generalisations I've been able to offer so far.
At an absolute minimum, just providing me with a pointer to the content of "the setup.py in question" would help. I'd probably only be able to say "that looks very unsupported, you need to fix that whether or not it's the cause of your problem", which you probably wouldn't find helpful, but at least it would give a starting point for me to understand what the heck you are doing here.
transparently executes the generated setup.py with the arguments passed to the original invocation of setup.py
That bit's not supported by setuptools (direct invocation of setup.py
) or by pip (even with the legacy code path, we don't guarantee that the arguments we pass to setup.py
are suitable for use in any other invocation).
I'm not trying to assign blame here, start a turf war or anything like that.
Understood. But I've been trying to offer helpful comments, even if they are suggestions that it's your code that needs to be fixed, and you've not taken any of them up, but have kept insisting that this is a "regression" in pip - frankly, without any clear evidence that this is the case (just to be 100% clear, it's not a regression just because some behaviour has changed - it could be a deliberate change as I believe this case is, or simply something that was never guaranteed to have a particular behaviour in the first place). From my perspective, it sure feels like blame 🙁
Anyway, I hope the above is of some use. If you can clarify your problem along the lines of what I've asked for above, I'll see if I can help further. But otherwise, I'm out of ideas and I'll drop the discussion at this point. Either way, best of luck working out what the problem is - it sounds like pybuilder is a pretty complicated package, and not easy to debug when things go wrong!
OK, this needs a self-contained reproducer that the maintainers can test with. Is that something that you can provide?
Culprit was found and it is bizarre and non-trivial.
The cause of it is sys.exit(exit_code)
at the end of the setup.py
here even if exit_code
is 0
here: https://github.com/pybuilder/pybuilder/blob/master/setup.py#L54
I'll provide a reproducible case.
Here's the difference in the outgoing trace that is installed in the pyproject_hooks/_in_process.py
as follows:
def get_requires_for_build_wheel(config_settings):
"""Invoke the optional get_requires_for_build_wheel hook
Returns [] if the hook is not defined.
"""
backend = _build_backend()
try:
hook = backend.get_requires_for_build_wheel
except AttributeError:
return []
else:
print(hook, config_settings)
import trace
t = trace.Trace(trace=1)
return t.runfunc(hook, config_settings)
$ diff -Nurd --color -b10 bad_exit.txt good_exit.txt
--- bad_exit.txt 2023-07-07 00:00:43.033580092 -0400
+++ good_exit.txt 2023-07-07 00:00:55.709571023 -0400
@@ -102,23 +102,18 @@
subprocess.py(1998): return self.returncode
subprocess.py(1131): if self.returncode is None and _active is not None:
subprocess.py(409): if retcode:
subprocess.py(414): return 0
<string>(64): --- modulename: setup, funcname: log
<string>(37): <string>(65): build_meta.py(495): sys.path[:] = sys_path
build_meta.py(496): sys.argv[0] = sys_argv_0
build_meta.py(322): with Distribution.patch():
--- modulename: contextlib, funcname: __exit__
contextlib.py(142): if typ is None:
- contextlib.py(150): if value is None:
- contextlib.py(154): try:
- contextlib.py(155): self.gen.throw(typ, value, traceback)
+ contextlib.py(143): try:
+ contextlib.py(144): next(self.gen)
--- modulename: build_meta, funcname: patch
build_meta.py(89): distutils.core.Distribution = orig
- contextlib.py(156): except StopIteration as exc:
- contextlib.py(161): except RuntimeError as exc:
- contextlib.py(179): except BaseException as exc:
- contextlib.py(186): if exc is not value:
- contextlib.py(188): exc.__traceback__ = traceback
- contextlib.py(189): return False
- build_meta.py(324): except SetupRequirementsError as e:
+ contextlib.py(145): except StopIteration:
+ contextlib.py(146): return False
+ build_meta.py(327): return requirements
Getting requirements to build wheel ... done
I suspect that ANY setup.py that uses sys.exit
will generate a problem because SystemExit
exception isn't handled to generate output for output.json
. But I'll need to create a reproducer for you to confirm that.
Unfortunately I'm flying tomorrow so unless the information above is not enough I'll get back to this in a couple of days.
I think this is a pretty good confirmation:
Handling SystemExit
like this solves the problem, although naturally does not return the required dependencies in case of a successful exit.
I guess there has to be more handling of sys.exit
somewhere around https://github.com/pypa/setuptools/blob/main/setuptools/build_meta.py#L340 as well to capture the dependencies.
def get_requires_for_build_wheel(config_settings):
"""Invoke the optional get_requires_for_build_wheel hook
Returns [] if the hook is not defined.
"""
backend = _build_backend()
try:
hook = backend.get_requires_for_build_wheel
except AttributeError:
return []
else:
print(hook, config_settings)
import trace
t = trace.Trace(trace=1)
try:
return t.runfunc(hook, config_settings)
except SystemExit as e:
if not e.code:
return []
raise
That said, one has to assume that sys.exit
may be used by a setup.py
even with a zero error code.
This seems to be a minimal reproducer. Save this setup.py
in a folder by itself:
from setuptools import setup
import sys
setup(
name="pep517-test-setup-py-support",
version="1.0",
)
sys.exit(0)
Then run pip wheel path/to/folder/
. I've just done this with pip 23.1.2, and I get the same failure as above:
Processing ./tests/samples/setup-py
Installing build dependencies ... done
Getting requirements to build wheel ... done
...
File "/home/takluyver/.local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
return self._call_hook('get_requires_for_build_wheel', {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/takluyver/.local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 317, in _call_hook
data = read_json(pjoin(td, 'output.json'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/takluyver/.local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 19, in read_json
with open(path, encoding='utf-8') as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpqgj1isss/output.json'
python -m build path/to/folder/
also fails on the example above; that gives less frontend complexity for investigating it in the backend.
I guess this could be solved inside setuptools though: https://github.com/pypa/setuptools/blob/main/setuptools/build_meta.py#L340.
I'll see if they'll be interested in uptaking the patch.
I'm just writing an issue for them
Here's the issue: https://github.com/pypa/setuptools/issues/3973
I agree that that looks like the place to fix it - feel free to have a go at a PR (but I'm not involved in setuptools, so they may want to approach it differently :shrug: )
FWIW, we could also catch BaseException
here.
Fixed in pypa/setuptools#3973
For posterity, the reason you were likely seeing the behaviour change starting pip 23.1 is that pip 23.1 is when pip started using pyproject.toml-based builds in more contexts (namely: if wheel isn't installed in the global environment). This will invoke the setuptools build-backend, which itself runs setup.py egg_info
under the hood -- checking if setup.py egg_info is invoked isn't going to provide much context (a different piece invokes it and, as the fix indicated, the change needed to happen on setuptools' end to not bubble up the error).
I'm incorporating by reference this PIP issue: https://github.com/pypa/pip/issues/12131
Since this is a pure
setup.py
there is nopyproject.toml
involved and legacy should keep functioning.