Open 0-wiz-0 opened 2 years ago
A PR to fix this would be welcome. It should be noted, however, that NetBSD is not a supported platform so a reproducer for this on a supported platform would also be helpful here.
@pfmoore I don't know the code base, can you please give me some pointers where this could be added? Btw, when you follow the first link above, you'll see that e.g. Fedora will also be affected. I don't have a Fedora system available though.
Class BuildEnvironment
in src/pip/_internal/build_env.py
would be the place to look. You'll need to be able to reproduce the issue on supported platforms in order to create a test, which will be part of creating a PR (the CI will provide the necessary environments, you "just" need to write suitable test code).
Here's the shell equivalent of what needs to happen:
cp ${PREFIX}/lib/python${PYVERSION}/site-packages/_distutils_system_mod.py ${OVERLAY}/lib/python${PYVERSION}/site-packages || true
Can you please add this in the appropriate python code? I don't have access to Fedora, and I don't know the code base.
@0-wiz-0 it works on fedora with overridden _sysconfig_name_tmpl
If you could provide a way to reproduce this, on a Linux system (eg: Fedora, RHEL, Ubuntu, Debian etc) that'd be great.
This patch fixes the problem for me:
--- src/pip/_internal/build_env.py.orig 2021-10-22 15:15:54.000000000 +0000
+++ src/pip/_internal/build_env.py
@@ -9,6 +9,7 @@ import sys
import textwrap
import zipfile
from collections import OrderedDict
+from shutil import copy
from sysconfig import get_paths
from types import TracebackType
from typing import TYPE_CHECKING, Iterable, Iterator, List, Optional, Set, Tuple, Type
@@ -93,6 +94,9 @@ class BuildEnvironment:
self._site_dir = os.path.join(temp_dir.path, "site")
if not os.path.exists(self._site_dir):
os.mkdir(self._site_dir)
+ distutils_distro_config = get_paths()["platlib"] + "/_distutils_system_mod.py"
+ if os.path.exists(distutils_distro_config):
+ copy(distutils_distro_config, self._site_dir)
with open(os.path.join(self._site_dir, "sitecustomize.py"), "w") as fp:
fp.write(
textwrap.dedent(
build_env.py
stopped importing syconfig, so the patch is now:
--- src/pip/_internal/build_env.py.orig 2023-02-17 18:31:10.000000000 +0000
+++ src/pip/_internal/build_env.py
@@ -8,6 +8,8 @@ import site
import sys
import textwrap
from collections import OrderedDict
+from shutil import copy
+from sysconfig import get_paths
from types import TracebackType
from typing import TYPE_CHECKING, Iterable, List, Optional, Set, Tuple, Type, Union
@@ -102,6 +104,9 @@ class BuildEnvironment:
self._site_dir = os.path.join(temp_dir.path, "site")
if not os.path.exists(self._site_dir):
os.mkdir(self._site_dir)
+ distutils_distro_config = get_paths()["platlib"] + "/_distutils_system_mod.py"
+ if os.path.exists(distutils_distro_config):
+ copy(distutils_distro_config, self._site_dir)
with open(
os.path.join(self._site_dir, "sitecustomize.py"), "w", encoding="utf-8"
) as fp:
Why does the file need to be copied? The point of having an isolated environment is so it has a fresh set of build tools and does not need to rely on the outside environment. This is conceptually against the grains of environment isolation.
pkgsrc has patches for Python to handle different operating systems the same way, to make the list of installed files the same. This is a property of Python itself on platforms using pkgsrc.
distutils needs overrides for this to work, since distutils does not use sysconfig
but its own version, distutils.sysconfig
.
More details can be found here and here.
This sounds like something pip should not do. Unless the distro override is standardised, there’s no reason why pip needs to implement a special treatment for a setuptools-specific mechanism. Environment isolation and PEP 517 is designed exactly to abstract away build tools specifics in the first place.
I think I agree with @uranusjr here. The linked setuptools issue describes the override as a short term solution. It also appears that there is a similar request for virtualenv. Does the stdlib venv module need this too? That’s a lot of work to handle a short term problem in setuptools.
I think that either the mechanism being used needs to be standardised, or setuptools needs to come up with a fix that doesn’t require multiple other tools to add special cases for it.
@jaraco writes about a long-term solution in distutils, true, but as far as I understand, distutils development is dead, so that will not be coming. Perhaps they can chime in about distutils development and if that change will ever come?
@jaraco also writes that pip previously copied distutils, so copying one file instead is less work and equivalent.
Distutils development is alive and well insofar as it’s used as a basis for Setuptools. Other uses are discouraged but still supported.
Description
setuptools recently added a distro-config file for distutils (see e.g. https://github.com/pypa/setuptools/pull/2896). pkgsrc uses this feature. pip does not copy this file to its overlay directory, causing 'pip install' to fail. 'pip install --no-build-isolation' works.
Example output of the error:
(This particular error happens because the pkgsrc distro config file overrides
_sysconfig_name_tmpl
to_sysconfigdata_${platform}
.)Expected behavior
pip should provide the
_distutils_system_mod.py
in its overlay directory, if the file exists.pip version
21.3.1
Python version
3.10.2
OS
NetBSD
How to Reproduce
pip install 'PyYAML<6'
though probably any package works.Output
No response
Code of Conduct