Open jaraco opened 4 weeks ago
The tests are failing with the "Pre-build distributions for test" part of the CI environment with the following traceback while attempting to invoke the PEP 517 build_wheel
hook:
Traceback (most recent call last):
File "/opt/pipx/.cache/8eea6f7f3e2f816/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/opt/pipx/.cache/8eea6f7f3e2f816/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/opt/pipx/.cache/8eea6f7f3e2f816/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
return _build_backend().build_wheel(wheel_directory, config_settings,
File "/tmp/build-via-sdist-d6kmbh6a/setuptools-69.5.1.post20240510/setuptools/build_meta.py", line 410, in build_wheel
return self._build_with_temp_dir(
File "/tmp/build-via-sdist-d6kmbh6a/setuptools-69.5.1.post20240510/setuptools/build_meta.py", line 395, in _build_with_temp_dir
self.run_setup()
File "/tmp/build-via-sdist-d6kmbh6a/setuptools-69.5.1.post20240510/setuptools/build_meta.py", line 311, in run_setup
exec(code, locals())
File "<string>", line 93, in <module>
File "/tmp/build-via-sdist-d6kmbh6a/setuptools-69.5.1.post20240510/setuptools/__init__.py", line 103, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.10/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.10/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/tmp/build-via-sdist-d6kmbh6a/setuptools-69.5.1.post20240510/setuptools/dist.py", line 968, in run_command
super().run_command(command)
File "/usr/lib/python3.10/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/tmp/build-env-_b1qxjv5/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 370, in run
install = self.reinitialize_command("install", reinit_subcommands=True)
File "/tmp/build-via-sdist-d6kmbh6a/setuptools-69.5.1.post20240510/setuptools/__init__.py", line 220, in reinitialize_command
cmd = _Command.reinitialize_command(self, command, reinit_subcommands)
File "/usr/lib/python3.10/distutils/cmd.py", line 305, in reinitialize_command
return self.distribution.reinitialize_command(command,
File "/usr/lib/python3.10/distutils/dist.py", line 938, in reinitialize_command
command = self.get_command_obj(command_name)
File "/usr/lib/python3.10/distutils/dist.py", line 858, in get_command_obj
cmd_obj = self.command_obj[command] = klass(self)
File "/tmp/build-via-sdist-d6kmbh6a/setuptools-69.5.1.post20240510/setuptools/__init__.py", line 178, in __init__
super().__init__(dist)
File "/usr/lib/python3.10/distutils/cmd.py", line 62, in __init__
self.initialize_options()
File "<string>", line 67, in initialize_options
File "/tmp/build-via-sdist-d6kmbh6a/setuptools-69.5.1.post20240510/setuptools/command/install.py", line 52, in initialize_options
orig.install.initialize_options(self)
File "/usr/lib/python3.10/_distutils_system_mod.py", line 33, in initialize_options
super().initialize_options()
TypeError: super(type, obj): obj must be an instance or subtype of type
I'm unable to replicate the failure in a clean docker image:
Aah. The failures are in the SETUPTOOLS_USE_DISTUTILS=stdlib
variant.
Unfortunately, I still can't replicate the issue with docker run -it jaraco/multipy-tox bash -c 'git clone https://github.com/pypa/setuptools; cd setuptools; git checkout feature/pep-621; env SETUPTOOLS_USE_DISTUTILS=stdlib pipx run --python python3.10 --pip-args "pyproject-hooks!=1.1" build'
Even running the action using act --job test --matrix python:3.10 --matrix platform:ubuntu-latest --matrix distutils:stdlib --container-architecture linux/amd64
, the tests run. Something is peculiar about how the tests are running in GitHub.
As a shot in the dark, I tried setting the SETUPTOOLS_USE_DISTUTILS explicitly in the script and that does replicate the failure when using the act
command above. It seems that somehow the --matrix distutils:stdlib
isn't having the intended effect.
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index ec2e567a1..cdc66d2bb 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -96,7 +96,7 @@ jobs:
run: |
rm -rf dist
# workaround for pypa/setuptools#4333
- pipx run --pip-args 'pyproject-hooks!=1.1' build
+ env SETUPTOOLS_USE_DISTUTILS=stdlib pipx run --pip-args 'pyproject-hooks!=1.1' build
echo "PRE_BUILT_SETUPTOOLS_SDIST=$(ls dist/*.tar.gz)" >> $GITHUB_ENV
echo "PRE_BUILT_SETUPTOOLS_WHEEL=$(ls dist/*.whl)" >> $GITHUB_ENV
rm -rf setuptools.egg-info # Avoid interfering with the other tests
After working around a bug in act, I've been able to replicate the failure reliably with the act command above.
Moreover, by rolling back to an older docker image using Ubuntu Jammy, I'm able to replicate the failure in a standalone docker image:
docker run -it jaraco/multipy-tox:jammy bash -c git clone https://github.com/pypa/setuptools; cd setuptools; git checkout feature/pep-621; env SETUPTOOLS_USE_DISTUTILS=stdlib pipx run --python python3.10 --pip-args "pyproject-hooks!=1.1" build
I've bisected the config from the bottom up and only after re-adding the full metadata section does the problem go away.
I'm not yet sure if it's related, but this code is only hit when the pyproject.toml config is present.
Any chance the cause of the problem is a late/delayed execution of the monkey patching in
Explaining a bit better: I was wondering if it is possible that the following is happening?
setuptools.command.install.install
is defined, orig.install
points to the original implementation in distutils/command/install.py
.__init__
method is executed, orig.install
actually points to _distutils_system_mod.install
. That would explain why the patched orig.install
can no longer be found in the __mro__
...But if it is a "late change of the binding" in orig.install
is causing this problem, it would mean that _distutils_system_mod
is being imported too late, and I don't know why that would be the case[^1]...
Possible workarounds we can think of (if the supposition above is true):
We can change all uses of orig.install
in setuptools/command/install.py
's methods to super()
, so it does not attempt to resolve the orig.install
binding and just get the right thing in the __mro__
.
Alternatively/similarly, we can try to avoid the "late binding resolution" in orig.install
by replacing:
- import distutils.command.install as orig
+ from distutils.command.install import install as orig_install
and then change all occurrences of orig.install => orig_install
. This way orig_install
inside the methods are guaranteed to point to the same object used during the class definition.
(that would prevent the error, but I suppose it would make the debian patching ineffective because it happens too late).
[^1]: And why that only happens when pyproject.toml
is used for configuration.
BTW, we had another instance reported in the past of a very similar exception in https://github.com/pypa/setuptools/issues/4136:
File "/usr/lib/python3.10/_distutils_system_mod.py", line 33, in initialize_options
super().initialize_options()
TypeError: super(type, obj): obj must be an instance or subtype of type
Investigating a bit more, I've put a breakpoint at the top of /usr/lib/python3.10/_distutils_system_mod.py
and I've found that when running a build against main, that breakpoint is never hit, but in this branch, the breakpoint is hit, so it seems that the _distutils_system_mod
is somehow not taking effect when the config is in setup.cfg
.
And the behavior on main seems like the right behavior - if SETUPTOOLS_USE_DISTUTILS=stdlib, then the _distutils_system_mod shouldn't be invoked.
Using this dockerfile:
FROM jaraco/multipy-tox:jammy
RUN apt install -y vim
RUN python3.10 -m pip install wheel build "pyproject-hooks!=1.1"
ENV SETUPTOOLS_USE_DISTUTILS=stdlib
RUN sed -i -e "s/import os/breakpoint()\nimport os/" /usr/lib/python3.10/_distutils_system_mod.py
workdir setuptools
CMD python3.10 -m build --no-isolation
And running it with the branch checked out locally, I can see the stack trace when _distutils_system_mod is imported during the build wheel step (and others):
It seems as if _apply_pyproject_toml
is causing setuptools' entry points to be loaded, which is somehow triggering the loading of setuptools._distutils
. Oh! Because Setuptools in some cases unconditionally imports setuptools._distutils
(even when SETUPTOOLS_USE_DISTUTILS=stdlib
).
But why then is that not an issue when using setup.cfg? There are several places in the code where setuptools._distutils
is imported. Are none of those hit when building under setup.cfg, and if so, why?
I put a breakpoint in setuptools._distutils
and confirmed that it's not imported at all during a full build under setup.cfg, so it's merely the fact that pyproject.toml requires extra evaluation of entry points that triggers this behavior.
- We can change all uses of
orig.install
insetuptools/command/install.py
's methods tosuper()
, so it does not attempt to resolve theorig.install
binding and just get the right thing in the__mro__
.
This approach seems most sound. I'll try that.
- We can change all uses of
orig.install
insetuptools/command/install.py
's methods tosuper()
, so it does not attempt to resolve theorig.install
binding and just get the right thing in the__mro__
.This approach seems most sound. I'll try that.
That does seem to fix the issue. I'll apply that in a separate PR against #4136.
Looks like the tests now pass when targeting main after #4361.
It's not immediately obvious to me why this change breaks the build.