[BUG] On Debian, install fails with "no such option 'install_layout'".

setuptools version

60.0.4

Python version

Python 3.9.9

OS

Ubuntu 21.10

Additional environment information

Running DEB packaging with dh-python.

Description

In Debian setuptools packaging an additional patch applied: https://sources.debian.org/src/python-setuptools/44.1.1-1/debian/patches/install-layout.diff

During 59.8.0, running debuild -uc -us for building setuptools WITHOUT about patch keep functioning.

During 60.0.0+, no matter with / without about patch, running debuild -uc -us keep failing.

If modify the build script as below, build now OK:

 override_dh_auto_install:
-       dh_auto_install
+       export SETUPTOOLS_USE_DISTUTILS=stdlib && dh_auto_install

Expected behavior

For v59.8.0 WITHOUT any additional patch it is OK:

sudo podman run -ti --rm ubuntu:21.10 bash
apt update
apt -y full-upgrade
apt -y install git curl wget python3 python3-all python3-dev dh-python fdupes build-essential devscripts debhelper
git clone -b alvistack/v59.8.0 https://github.com/alvistack/pypa-setuptools.git
cd pypa-setuptools
git clean -xdf
tar zcvf ../python-setuptools_59.8.0.orig.tar.gz --exclude=.git .
debuild -uc -us

For v60.0.4 WITH SETUPTOOLS_USE_DISTUTILS=stdlib it is OK:

sudo podman run -ti --rm ubuntu:21.10 bash
apt update
apt -y full-upgrade
apt -y install git curl wget python3 python3-all python3-dev dh-python fdupes build-essential devscripts debhelper
git clone -b alvistack/v60.0.4 https://github.com/alvistack/pypa-setuptools.git
cd pypa-setuptools
git clean -xdf
tar zcvf ../python-setuptools_60.0.4.orig.tar.gz --exclude=.git .
debuild -uc -us

How to Reproduce

For v60.0.4 WITHOUT SETUPTOOLS_USE_DISTUTILS=stdlib it is FAILED:

sudo podman run -ti --rm ubuntu:21.10 bash
apt update
apt -y full-upgrade
apt -y install git curl wget python3 python3-all python3-dev dh-python fdupes build-essential devscripts debhelper
git clone -b alvistack/v60.0.4 https://github.com/alvistack/pypa-setuptools.git
cd pypa-setuptools
find *.spec debian/rules -type f | xargs sed -i 's/export SETUPTOOLS_USE_DISTUTILS=stdlib && //g'
git clean -xdf
tar zcvf ../python-setuptools_60.0.4.orig.tar.gz --exclude=.git .
debuild -uc -us

Output

   debian/rules override_dh_auto_install
make[1]: Entering directory '/pypa-setuptools'
dh_auto_install
I: pybuild base:232: /usr/bin/python3 setup.py install --root /pypa-setuptools/debian/tmp 
/pypa-setuptools/setuptools/dist.py:723: UserWarning: Usage of dash-separated 'build-lib' will not be supported in future versions. Please use the underscore name 'build_lib' instead
  warnings.warn(
/pypa-setuptools/setuptools/dist.py:723: UserWarning: Usage of dash-separated 'install-layout' will not be supported in future versions. Please use the underscore name 'install_layout' instead
  warnings.warn(
/pypa-setuptools/setuptools/dist.py:723: UserWarning: Usage of dash-separated 'install-scripts' will not be supported in future versions. Please use the underscore name 'install_scripts' instead
  warnings.warn(
/pypa-setuptools/setuptools/dist.py:723: UserWarning: Usage of dash-separated 'install-lib' will not be supported in future versions. Please use the underscore name 'install_lib' instead
  warnings.warn(
running install
/pypa-setuptools/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
error: error in /pypa-setuptools/.pybuild/cpython3_3.9/.pydistutils.cfg: command 'install_with_pth' has no such option 'install_layout'
E: pybuild pybuild:353: install: plugin distutils failed with: exit code=1: /usr/bin/python3 setup.py install --root /pypa-setuptools/debian/tmp 
dh_auto_install: error: pybuild --install -i python{version} -p 3.9 --dest-dir /pypa-setuptools/debian/tmp returned exit code 13
make[1]: *** [debian/rules:6: override_dh_auto_install] Error 25
make[1]: Leaving directory '/pypa-setuptools'
make: *** [debian/rules:16: binary] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
debuild: fatal error at line 1182:
dpkg-buildpackage -us -uc -ui failed

Code of Conduct

[X] I agree to follow the PSF Code of Conduct

In https://github.com/pypa/distutils/pull/68, I developed the most extensive hack to support Debian's status quo. In that hack, distutils searches for _distutils_system_mod (a top-level module) to allow it to apply patches to distutils as found in Setuptools. I drafted a possible implementation of that hook does include support for --install-layout, but I haven't had any response from the Debian maintainers. Perhaps you could help testing by adding that file to somewhere on sys.path and see if that corrects the behavior?

In pypa/distutils#68, I developed the most extensive hack to support Debian's status quo. In that hack, distutils searches for _distutils_system_mod (a top-level module) to allow it to apply patches to distutils as found in Setuptools. I drafted a possible implementation of that hook does include support for --install-layout, but I haven't had any response from the Debian maintainers. Perhaps you could help testing by adding that file to somewhere on sys.path and see if that corrects the behavior?

Ummm... Sorry not good enough:

sudo podman run -ti --rm ubuntu:21.10 bash
apt update
apt -y full-upgrade
apt -y install git curl wget python3 python3-all python3-dev dh-python fdupes build-essential devscripts debhelper
git clone -b alvistack/v60.0.4 https://github.com/alvistack/pypa-setuptools.git
cd pypa-setuptools
find *.spec debian/rules -type f | xargs sed -i 's/export SETUPTOOLS_USE_DISTUTILS=stdlib && //g'
curl -skL https://raw.githubusercontent.com/pypa/distutils/115da201da0e648d003dabbbd597d285d0d271f2/_distutils_system_mod.py > /usr/lib/python3/dist-packages/_distutils_system_mod.py
git clean -xdf
tar zcvf ../python-setuptools_60.0.4.orig.tar.gz --exclude=.git .
debuild -uc -us

Result with error message KeyError: 'deb_system':

/pypa-setuptools/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
Traceback (most recent call last):
  File "/pypa-setuptools/setup.py", line 87, in <module>
    dist = setuptools.setup(**setup_params)
  File "/pypa-setuptools/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/pypa-setuptools/setuptools/_distutils/core.py", line 148, in setup
    return run_commands(dist)
  File "/pypa-setuptools/setuptools/_distutils/core.py", line 163, in run_commands
    dist.run_commands()
  File "/pypa-setuptools/setuptools/_distutils/dist.py", line 967, in run_commands
    self.run_command(cmd)
  File "/pypa-setuptools/setuptools/_distutils/dist.py", line 985, in run_command
    cmd_obj.ensure_finalized()
  File "/pypa-setuptools/setuptools/_distutils/cmd.py", line 107, in ensure_finalized
    self.finalize_options()
  File "/pypa-setuptools/setup.py", line 66, in finalize_options
    install.finalize_options(self)
  File "/pypa-setuptools/setuptools/command/install.py", line 45, in finalize_options
    orig.install.finalize_options(self)
  File "/pypa-setuptools/setuptools/_distutils/command/install.py", line 366, in finalize_options
    self.finalize_unix()
  File "/usr/lib/python3/dist-packages/_distutils_system_mod.py", line 41, in finalize_unix
    self.select_scheme("deb_system")
  File "/pypa-setuptools/setuptools/_distutils/command/install.py", line 565, in select_scheme
    _select_scheme(self, name)
  File "/pypa-setuptools/setuptools/_distutils/command/install.py", line 127, in _select_scheme
    vars(ob).update(_remove_set(ob, _scheme_attrs(_resolve_scheme(name))))
  File "/pypa-setuptools/setuptools/_distutils/command/install.py", line 152, in _scheme_attrs
    scheme = _load_schemes()[name]
KeyError: 'deb_system'
E: pybuild pybuild:353: install: plugin distutils failed with: exit code=1: /usr/bin/python3 setup.py install --root /pypa-setuptools/debian/tmp 
dh_auto_install: error: pybuild --install -i python{version} -p 3.9 --dest-dir /pypa-setuptools/debian/tmp returned exit code 13
make[1]: *** [debian/rules:6: override_dh_auto_install] Error 25
make[1]: Leaving directory '/pypa-setuptools'
make: *** [debian/rules:16: binary] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
debuild: fatal error at line 1182:
dpkg-buildpackage -us -uc -ui failed

The way the example Debian _distutils_system_mod is defined, it's expecting a deb_system scheme to be present in sysconfig._INSTALL_SCHEMES, so you'll also need to monkeypatch sysconfig to ensure that scheme exists.

@jaraco I had read though these issues and generally understand your concern:

First of all I am not an official Debian packager for setuptools, just on behalf of my personal cross OS packaging project, target for Ansible installation in deb/rpm:

IMHO, possible solution could be:

Keep the existing Debian-specific patching style, maintain by Debian packager, no additional support from setuptools with monkeypatch
- Keep it simple stupid
- Keep maintenance duty outside setuptools
- Who need this dirty hack, who maintenance it ;-)
- BTW, seems 60.0.0+ block this approach?
Provide a global on/off switch DURING setuptools installation, e.g.
- Once setuptools installed with SETUPTOOLS_USE_DISTUTILS=stdlib, any else on going package installation will also operate with SETUPTOOLS_USE_DISTUTILS=stdlib by default
- Else, due to 60.0.0+ changes, I need to manually add SETUPTOOLS_USE_DISTUTILS=stdlib for all OBS packaging script
Monkeypatch style
- Complicated
- Both setuptools and Debian maintainer need to work together
- Increase maintenance difficulties
Support --install-layout officially inside setuptools
- Keep it simple stupid for single maintenance party

Personally, I vote for 4 >> 2 >> 1 >> 3

4. Support --install-layout officially inside setuptools

I considered this approach, but there are two problems with the approach:

The design is Debian-specific and applies Debian-specific behaviors. It was designed with a very narrow scope and doesn't consider other platforms. If the feature can be shown to have general-purpose benefit and be submitted as a PR to this project, that may be acceptable.
The design runs in conflict with the general-purpose approach, which has been coordinated with other platforms and which is available to other installer tools (flit, build, pip) via sysconfig (install schemes and get_preferred_scheme). It's imperative that Debian utilize the general-purpose approach and if it's inadequate, describe what the gaps are and propose a solution that fills those gaps.

2. Provide a global on/off switch DURING setuptools installation, e.g.

Remember that SETUPTOOLS_USE_DISTUTILS=stdlib is a temporary escape hatch. It's there to enable end users and installers to opt out of the inevitable transition to a world without distutils in the stdlib (slated for Python 3.12). Making this change permanent in an environment does simplify orchestration of that environment, but it also has the following downsides:

Users can no longer experiment with installing their packages with =local.
The platform becomes pinned to a deprecated but comfortable state as the deadline for hard breakage approaches.
Semantic expectations set for the purpose of SETUPTOOLS_USE_DISTUTILS will change after over a year of stability. Perhaps a different flag could be exposed to selectively install the functionality.
It would create divergent installs of Setuptools. So a given Setuptools version would have different behaviors depending on what environment variables were set at install time, multiplying the complexity of triage for any issue.

Else, due to 60.0.0+ changes, I need to manually add SETUPTOOLS_USE_DISTUTILS=stdlib for all OBS packaging script

Could you set that environment variable globally in the Debian environment? At least for now until Debian can adopt an approach that no longer relies on monkeypatching?

1. Keep the existing Debian-specific patching style

As distutils is going away, this approach is untenable. Setuptools is committed to adopting distutils, so if Debian wishes to own a patching approach, they would need to patch setuptools. They're welcome to do so if that's what they wish, but Setuptools is expecting to perform a great deal of refactoring to properly adopt distutils and remove the cruft around it, so maintaining a monkeypatch for versions of Setuptools as its evolving is going to be burdensome. That's why this project is going out of its way to provide reliable if temporary interfaces to allow for the customizations currently provided by patches.

3. Monkeypatch style

I'm guessing by "Monkeypatch style", you mean the approach currently proposed by this project?

Yes, this approach is somewhat complicated, but it's less brittle than patching as done in option 1 because it specifically patches the runtime objects and not the code, so only relies on the hook points provided by distutils. Yes, it requires coordination, but since it's meant to be temporary, I'd expect that Debian would find a way to move away from relying on these interfaces.

The primary advantage of the monkeypatch approach is that it keeps the patched behavior largely disentangled from the default implementation. That is, any Python programmer can see the entirety of the patch and disable it by deleting _distutils_system_mod or commenting out the apply_customizations() call.

I do see your point about maybe preferring option 1. I might prefer that too. Unfortunately, I don't think Debian is interested in owning it, so I've supplied the system mod approach as a means of unblocking users while Debian figures out how they want to proceed. If they want to keep these behaviors long-term, they should propose the behaviors as a proper feature of the project (with designs, implementations, tests, etc).

I am not an official Debian packager for setuptools

I do regret that you've been caught up in this. I would have preferred that Debian maintainers would have picked this up sooner and implemented the recommended patch, tested it, and provided forward guidance to downstream packagers like yourself. I appreciate your feedback and the effort you've taken to understand the issue.

I welcome you or others to engage with the Debian project to devise a better solution. Right now, _distutils_system_mod is my best proposal, but it's optional and I won't be offended if Debian wishes to take another approach, and I offer my time and services as maintainer of Setuptools/distutils to give feedback or otherwise help guide their decisions.

It does feel like you're 🤏 this close to having a working solution. Let me take a stab at extending the _distutils_system_mod to provide the deb_system and unix_system install schemes needed to honor the --install-layout. Thanks for the repro, which should make it easy for me to test.

In pypa/distutils@69f85733543b9f6257e89c4f79f9507f20d06b49 (debian-patch branch), I've added one additional step that extends the install schemes in sysconfig to include the deb_system. And now the repro above builds without error:

$ cat Dockerfile
FROM ubuntu:21.10
RUN apt update
RUN apt -y full-upgrade
RUN DEBIAN_FRONTEND=noninteractive apt -y install git curl wget python3 python3-all python3-dev dh-python fdupes build-essential devscripts debhelper
RUN git clone -b alvistack/v60.0.4 https://github.com/alvistack/pypa-setuptools.git
WORKDIR pypa-setuptools
RUN find *.spec debian/rules -type f | xargs sed -i 's/export SETUPTOOLS_USE_DISTUTILS=stdlib && //g'
RUN curl -skL https://raw.githubusercontent.com/pypa/distutils/69f8573354/_distutils_system_mod.py > /usr/lib/python3/dist-packages/_distutils_system_mod.py
RUN git clean -xdf
RUN tar zcvf ../python-setuptools_60.0.4.orig.tar.gz --exclude=.git .
RUN debuild -uc -us
$ 
$ docker build .
[+] Building 73.5s (15/15) FINISHED                                                                                                            
 => [internal] load build definition from Dockerfile                                                                                      0.0s
 => => transferring dockerfile: 727B                                                                                                      0.0s
 => [internal] load .dockerignore                                                                                                         0.0s
 => => transferring context: 2B                                                                                                           0.0s
 => [internal] load metadata for docker.io/library/ubuntu:21.10                                                                           0.3s
 => [ 1/11] FROM docker.io/library/ubuntu:21.10@sha256:cc8f713078bfddfe9ace41e29eb73298f52b2c958ccacd1b376b9378e20906ef                   0.0s
 => CACHED [ 2/11] RUN apt update                                                                                                         0.0s
 => CACHED [ 3/11] RUN apt -y full-upgrade                                                                                                0.0s
 => [ 4/11] RUN DEBIAN_FRONTEND=noninteractive apt -y install git curl wget python3 python3-all python3-dev dh-python fdupes build-esse  47.6s
 => [ 5/11] RUN git clone -b alvistack/v60.0.4 https://github.com/alvistack/pypa-setuptools.git                                           6.8s 
 => [ 6/11] WORKDIR pypa-setuptools                                                                                                       0.0s 
 => [ 7/11] RUN find *.spec debian/rules -type f | xargs sed -i 's/export SETUPTOOLS_USE_DISTUTILS=stdlib && //g'                         0.2s 
 => [ 8/11] RUN curl -skL https://raw.githubusercontent.com/pypa/distutils/69f8573354/_distutils_system_mod.py > /usr/lib/python3/dist-p  0.6s 
 => [ 9/11] RUN git clean -xdf                                                                                                            0.2s 
 => [10/11] RUN tar zcvf ../python-setuptools_60.0.4.orig.tar.gz --exclude=.git .                                                         0.6s 
 => [11/11] RUN debuild -uc -us                                                                                                          10.8s
 => exporting to image                                                                                                                    6.2s 
 => => exporting layers                                                                                                                   6.2s 
 => => writing image sha256:e01c723814752a10875080257043a0e10e94076418d1f91c21ac34501d639cf1                                              0.0s

Is that sufficient to close this issue, at least until Debian maintainers step up with an alternative proposal?

In pypa/distutils@69f8573 (debian-patch branch), I've added one additional step that extends the install schemes in sysconfig to include the deb_system. And now the repro above builds without error:

@jaraco Thank you very much for the _distutils_system_mod.py update, so now my hack for packaging work much smoothly:

sudo podman run -ti --rm ubuntu:21.10 bash

# 1st build
apt update
apt -y full-upgrade
apt -y install git curl wget python3 python3-all python3-dev dh-python fdupes build-essential devscripts debhelper
git clone -b alvistack/v60.1.0 https://github.com/alvistack/pypa-setuptools.git
cd pypa-setuptools
git clean -xdf
tar zcvf ../python-setuptools_60.1.0.orig.tar.gz --exclude=.git .
debuild -uc -us

# Install the deb
dpkg -i ../*.deb
ls -la /usr/lib/python3/dist-packages/_distutils_system_mod.py

# 2nd build
git clean -xdf
tar zcvf ../python-setuptools_60.1.0.orig.tar.gz --exclude=.git .
debuild -uc -us

Moreover, I will try to ping Debian official python3-setuptools maintainer for this issue. They are already in 59.6.0 now (see https://packages.debian.org/sid/python3-setuptools) so soon or later they also need to face the changes ;-)

@jaraco I guess #2977 could be a good reason for us implementing dist-specific support inside setuptools, because Fedora 34+ also fail with something similar as this issue :-(

Somewhat related (just fyi): setuptools 60 defaulting to SETUPTOOLS_USE_DISTUTILS=local combined with https://github.com/pypa/pip/issues/6264 and Debian's patching of setuptools cause AttributeError: install_layout in --system-site-packages venvs. Reported as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1003252, although the primary cause is probably https://github.com/pypa/pip/issues/6264.

pypa / setuptools