search

pypa / setuptools

Official project repository for the Setuptools build system
https://pypi.org/project/setuptools/
MIT License
2.53k stars 1.19k forks source link

[BUG] PEP-660 editable installs breaks versioneer #3501

Closed Lyle-Alloy closed 2 years ago

Lyle-Alloy commented 2 years ago

setuptools version

49.2.1

Python version

3.9.2

OS

macOS 12.5

Additional environment information

versioneer.py is version 0.19 wheel is 0.37.1 (not sure if relevant)

Description

Doing an editable install of our code (with pip install -e .) on a github actions runner is failing on macOS. We use versioneer for version numbers and something changed that seems to be breaking it. Do you have any ideas on what might have changed that is making versioneer unhappy?

EDIT: This now fails on linux (Ubuntu 20.04) as well

Expected behavior

It does not fail, prior to this update it was fine.

How to Reproduce

we have a repository with versioneer 0.19, and the setup.py and pyproject.toml below:

setup.py NOTE: some details removed and replaced with `...` ```python import os from pathlib import Path import sys from setuptools import setup, find_packages import versioneer setup( name="Project", version=versioneer.get_version(), # from versioneer install guide cmdclass=versioneer.get_cmdclass(), # from versioneer install guide author="Lyle Cheatham", author_email="lyle@alloyenterprises.co", keywords="", packages=find_packages(exclude=["test"]), install_requires=[...], dependency_links=[...], include_package_data=True, python_requires="~=3.9.2", entry_points={ "console_scripts": [...], }, ) ```
pyproject.toml NOTE: some details replaced with `...` ```toml ######################################### BUILD ########################################## [build-system] requires = [ "setuptools>=49.2.1", "wheel~=0.37.1", "cython~=0.29.28", "gitpython~=3.1.26", ] ######################################### BLACK ########################################## [tool.black] ... ######################################### PYTEST ######################################### [tool.pytest.ini_options] ... ######################################### PYLINT ######################################### [tool.pylint] ... ```

Output

Log excerpt ```console × Building editable for Project (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [55 lines of output] running editable_wheel creating /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-wheel-55z7vnhm/tmpziqr33el/Project.egg-info writing /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-wheel-55z7vnhm/tmpziqr33el/Project.egg-info/PKG-INFO writing dependency_links to /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-wheel-55z7vnhm/tmpziqr33el/Project.egg-info/dependency_links.txt writing entry points to /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-wheel-55z7vnhm/tmpziqr33el/Project.egg-info/entry_points.txt writing requirements to /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-wheel-55z7vnhm/tmpziqr33el/Project.egg-info/requires.txt writing top-level names to /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-wheel-55z7vnhm/tmpziqr33el/Project.egg-info/top_level.txt writing manifest file '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-wheel-55z7vnhm/tmpziqr33el/Project.egg-info/SOURCES.txt' reading manifest file '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-wheel-55z7vnhm/tmpziqr33el/Project.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-wheel-55z7vnhm/tmpziqr33el/Project.egg-info/SOURCES.txt' creating '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-wheel-55z7vnhm/tmpziqr33el/Project-0.20.1+22.gdfe7c96.dist-info' creating /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-wheel-55z7vnhm/tmpziqr33el/Project-0.20.1+22.gdfe7c96.dist-info/WHEEL /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-7altab77/overlay/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( running build running build_py Traceback (most recent call last): File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-7altab77/overlay/lib/python3.9/site-packages/setuptools/command/editable_wheel.py", line 138, in run self._create_wheel_file(bdist_wheel) File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-7altab77/overlay/lib/python3.9/site-packages/setuptools/command/editable_wheel.py", line 284, in _create_wheel_file files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp) File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-7altab77/overlay/lib/python3.9/site-packages/setuptools/command/editable_wheel.py", line 257, in _run_build_commands self.run_command("build") File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-7altab77/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command self.distribution.run_command(command) File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-7altab77/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command super().run_command(command) File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-7altab77/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command cmd_obj.run() File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-7altab77/overlay/lib/python3.9/site-packages/setuptools/command/build.py", line 33, in run super().run() File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-7altab77/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 132, in run self.run_command(cmd_name) File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-7altab77/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command self.distribution.run_command(command) File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-7altab77/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command super().run_command(command) File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-7altab77/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 992, in run_command cmd_obj.run() File "/Users/runner/work/project/project/versioneer.py", line 1568, in run write_to_version_file(target_versionfile, versions) File "/Users/runner/work/project/project/versioneer.py", line 1223, in write_to_version_file os.unlink(filename) FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpd49cxg_x.build-lib/project/_version.py' UPDATING /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpd49cxg_x.build-lib/project/_version.py error: Support for editable installs via PEP 660 was recently introduced in `setuptools`. If you are seeing this error, please report to: https://github.com/pypa/setuptools/issues Meanwhile you can try the legacy behavior by setting an environment variable and trying to install again: SETUPTOOLS_ENABLE_FEATURES="legacy-editable" [end of output] ```
synchronizing commented 2 years ago

Reported in #3498-comment. Related to #3499 and #3500.

Lyle-Alloy commented 2 years ago

Is this the same thing as #3498? I don't see it going down the same code path and hitting the typo'd attribute. Thanks for being so quick with the response though!

Also update on the Linux vs MacOS thing, that was a red herring. All of our linux builds are now also failing

synchronizing commented 2 years ago

It was a comment in the mix of #3498, but unrelated to the original issue - added it in there for a bit more context, since a contributor replied. All of our builds are failing as well, both MacOS and Linux.

abravalheri commented 2 years ago

Thank you very much guys, I am investigating both issues. I hope to get back to you soon.

abravalheri commented 2 years ago

Hi @Lyle-Alloy, maybe I will need a bit more detail on this one...

You reported that the problem also happens on Ubuntu 20.02 right? (Unfortunately I don't have a macOS available for testing).

I tried to use the following reproducer, but things seem to be working:

cd /tmp
rm -rf myproj
mkdir -p myproj/myproj
cd myproj
cat <<EOF > pyproject.toml
[build-system]
requires = [
    "setuptools>=49.2.1",
    "wheel~=0.37.1",
    "cython~=0.29.28",
    "gitpython~=3.1.26",
]

[tool.black]

[tool.pytest.ini_options]

[tool.pylint]
EOF

cat <<EOF > setup.py
import os
from pathlib import Path
import sys
from setuptools import setup, find_packages
import versioneer

setup(
    name="Project",
    version=versioneer.get_version(),  # from versioneer install guide
    cmdclass=versioneer.get_cmdclass(),  # from versioneer install guide
    author="Lyle Cheatham",
    author_email="lyle@alloyenterprises.co",
    keywords="",
    packages=find_packages(exclude=["test"]),
    install_requires=[],
    dependency_links=[],
    include_package_data=True,
    python_requires="~=3.9.2",
    entry_points={
        "console_scripts": [],
    },
)
EOF

echo "print(42)" > myproj/__init__.py
git init .
git add .

virtualenv -p py39 .venv
.venv/bin/python -m pip install 'versioneer==0.19' -U pip
.venv/bin/versioneer install

cat <<EOF > setup.cfg
[versioneer]
VCS = git
style = pep440
versionfile_source =
versionfile_build =
tag_prefix =
parentdir_prefix =
EOF

git add setup.cfg versioneer.py
git commit -m "Initial commit"
git tag 0.1

.venv/bin/python -m pip install -e .
# ...
# Successfully installed Project-0.1

I am currently trying to download a container for Python 3.9.2, but it is just taking ages... Meanwhile could you check if the reproducer above works in your machine? If it works what changes are necessary to make so it breaks with the same error you are seeing?

msarahan commented 2 years ago

I have the same original error. The reproducer above runs successfully for me. I will work on finding a modification that reproduces the error.

msarahan commented 2 years ago

To reproduce the issue, you must set the version files:

cat <<EOF > setup.cfg
[versioneer]
VCS = git
style = pep440
versionfile_source = _version.py
versionfile_build = _version.py
tag_prefix =
parentdir_prefix =
EOF
basnijholt commented 2 years ago

I am seeing this error too in Versioningit, even though @abravalheri added support for PEP660 in https://github.com/jwodder/versioningit/pull/38.

The solution will likely be similar.

abravalheri commented 2 years ago

Thank you very much @msarahan.

I am still trying to understand what is going on...

So far what I think is happening is the following:

I managed to get around the error in 2 different ways:

diff --git i/versioneer.py w/versioneer.py
index 1040c21..ec375ea 100644
--- i/versioneer.py
+++ w/versioneer.py
@@ -1220,7 +1220,8 @@ def versions_from_file(filename):

 def write_to_version_file(filename, versions):
     """Write the given version number to the given _version.py file."""
-    os.unlink(filename)
+    if os.path.exists(filename):
+        os.unlink(filename)
     contents = json.dumps(versions, sort_keys=True,
                           indent=1, separators=(",", ": "))
     with open(filename, "w") as f:

OR

diff --git i/versioneer.py w/versioneer.py
index 1040c21..5f57a8b 100644
--- i/versioneer.py
+++ w/versioneer.py
@@ -1559,6 +1559,8 @@ def get_cmdclass(cmdclass=None):
             cfg = get_config_from_root(root)
             versions = get_versions()
             _build_py.run(self)
+            if self.editable_mode:
+                return
             # now locate _version.py in the new build/ directory and replace
             # it with an updated value
             if cfg.versionfile_build:

I suspect the second patch makes more sense...

My conclusion is that solving this issue would require a bit of collaboration with versioneer. I will try to see if I can start proposing something, but I am not familiar with their code base.

abravalheri commented 2 years ago

@basnijholt thanks for reporting, please feel free to open a new issue with a reproducer.

msarahan commented 2 years ago

I am partial to your first suggestion - it's just better defensive programming around any issue. It also then doesn't introduce a dependency on having the editable_mode attribute.

Unfortunately, this is probably causing issues in more places. In particular, I am not using verisoneer, but rather miniver, which is a versioneer derivative. My point is that there is an unknown number of projects that may be breaking here. Instead of forcing a fix across all of them, is there some (admittedly inherently janky) thing we can do on the setuptools side? My first thought was to treat the source dir as the build dir, but that is definitely trouble (it would then be modifying source in possibly crazy ways as "setup"). Perhaps introduce a new cmdclass that does not copy, which is the new default, but for the old cmdclass that versioneer (and others) inherit/extend, do the source copying to the temporary folder, even though it isn't used? And post a warning?

abravalheri commented 2 years ago

Hi @msarahan thank you very much for the in-depth analysis and suggestion.

Unfortunately things are a bit more complicated...

Previously, the develop command special-cased build_py and build_ext and treated each command considering the knowledge on what they do. It ignored all the other build subcommands.

This was problematic because custom build steps could not run during an editable install.

editable_wheel tries to use a more generic approach and fix the problem of custom build steps:

I don't know exactly, but I think versioneer uses both assumptions:

I have to think a little more to understand if there is a different alternative that can fix this problem for a larger number of projects... Please feel free to keep the suggestions coming :P

(Any PR for fixing this problem directly in setuptools without compromising the new functionality is also very appreciated).

msarahan commented 2 years ago

The relevant breaking code in miniver is: https://github.com/jbweston/miniver/blob/master/miniver/_version.py#L152-L163

They have a try/except around removing the file, but the writing the file will also fail because the temp dir is not there. Any fixes would have to account for both. One thing that might help is if setuptools at least created that temp folder. It would silence the error for miniver, at least. I don't think that's actually the right thing, though. Just a hack.

I think you've done the right thing for the long-term here. It does suck to break many things, but I don't know that there is any satisfactory workaround. Thanks for your efforts.

abravalheri commented 2 years ago

@msarahan, thank you very much for the pointer.

I am exploring in https://github.com/pypa/setuptools/pull/3506 a solution to avoid crashing builds with a similar problem.

Please feel free to have a look.

One thing that might help is if setuptools at least created that temp folder.

I think the existing implementation will create the base directory (build_lib), but it will not create the parent directory for each file that is not going to be added...

effigies commented 2 years ago

Hi all, I've pushed versioneer 0.23 which should fix this. Please give it a try and report any issues you find.

jwodder commented 2 years ago

@abravalheri I'm the maintainer of versioningit, and I'm unclear under about how this bug affects my plugin. A PR was submitted to my project that allegedly fixes the problem, but no clear bug report or reproducing test case was provided, and the current test suite already passes without error using the latest setuptools. Apparently, the bug relates to the sdist command being invoked during editable installs? Does this happen, and, if so, under what conditions?

abravalheri commented 2 years ago

Hi @jwodder thank you for bringing this up, I will have a look on the report later or during the weekend.

Apparently, the bug relates to the sdist command being invoked during editable installs? Does this happen, and, if so, under what conditions?

In setuptools the (original) sdist command is used as a basis for the manifest_maker which is in turn used to find which files to include in the distribution or not. The editable mode might try to find out what are these files. I need to have a proper look in the issue to see what is happening, and I agree that without the proper reproducer it might be complicated.

The previous PR I provided targeted the reproduction described in https://discuss.python.org/t/help-testing-pep-660-support-in-setuptools/16904/35 and was confirmed to solve the OP.

abravalheri commented 2 years ago

I am about to merge the solution presented in https://github.com/pypa/setuptools/pull/3506 which should also solve the general problem reported here for older versions of versioneer and other packages that use the same approach.

I don't know if this will solve the problem @basnijholt is facing, but anyone is welcome to open a new PR if there is other circumstances that have not been handled (a reproducer really helps).