[BUG] Setuptools 69.0.3 `setup.py develop` generates a .egg-link file with underscores in the name component

[ichard26]

setuptools version

69.0.3

Python version

CPython 3.11.7

OS

Ubuntu 22.04.03 LTS

Additional environment information

No response

Description

Happy New Year! :tada:

I was debugging pip's CI which has been red ever since the release of Setuptools 69.0.3. Most of them are easily fixed by removing the underscore -> dash normalization assumption, however, there is one place where leaving the underscores intact causes issues: legacy editable installs.

As [documented in the Setuptools documentation](https://setuptools.pypa.io/en/latest/deprecated/python_eggs.html#:~:text=The%20%E2%80%9Cname%E2%80%9D%20and%20%E2%80%9Cversion%E2%80%9D%20should%20be%20escaped%20using%20the%20to_filename()%20function%20provided%20by%20pkg_resources%2C%20after%20first%20processing%20them%20with%20safe_name()%20and%20safe_version()%20respectively.), the distribution name part of egg filenames should be normalized by safe_name():

The “name” and “version” should be escaped using the to_filename() function provided by pkg_resources, after first processing them with safe_name() and safe_version() respectively. These latter two functions can also be used to later “unescape” these parts of the filename. (For a detailed description of these transformations, please see the “Parsing Utilities” section of the pkg_resources manual.)

Setuptools does not honour this.[^1] This is actually fine in most situations as far as I can tell since there are modern ways for pip to discover installed distributions that don't rely on eggs, but setup.py develop does not generate this modern metadata. Thus, pip falls back to searching sys.path for a .egg-link file to determine whether a distribution is editably installed. Pip assumes the egg link name will be normalized by safe_name() so this logic returns a false negative despite version_pkg being editably installed in fact.

If I'm being honest, I have no idea whose problem this is, but this does mean for projects that do not implement PEP 518 and have underscores in their name will not be recognized as editably installed. If this is better transferred to the pip repository, please let me know!

[^1]: Well, if you read the documentation closely, apparently you're supposed to pass the made safe name to pkg_resources.to_filename() which would turn the dash back into an underscore but setuptools does not seem to do this anyway /shrug

Expected behavior

See above.

How to Reproduce

0. Create an environment with pip<24, setuptools==69.0.3 and wheel.
1. Create a new directory for testing and create a setup.py representing a version_pkg:

# setup.py
import setuptools; setuptools.setup(name="version_pkg")

2. Run pip install -e . in the directory
3. Run pip freeze or pip list and observe that pip doesn't realize it's an editable install
4. Observe that the egg-link file has a underscore
5. Rerun this with Setuptools 69.0.2 and observe that pip can recognize that version_pkg is installed as an editable

Output

ichard26@asus-ubuntu:~/dev/oss/pip/temp/pip-test-package$ pip install setuptools==69.0.3 -q
ichard26@asus-ubuntu:~/dev/oss/pip/temp/pip-test-package$ python ../generate.py 
ichard26@asus-ubuntu:~/dev/oss/pip/temp/pip-test-package$ pip install -e .
Obtaining file:///home/ichard26/dev/oss/pip/temp/pip-test-package
  Preparing metadata (setup.py) ... done
Installing collected packages: version_pkg
  Running setup.py develop for version_pkg
Successfully installed version_pkg-0.1
ichard26@asus-ubuntu:~/dev/oss/pip/temp/pip-test-package$ pip freeze
version_pkg==0.1
ichard26@asus-ubuntu:~/dev/oss/pip/temp/pip-test-package$ pip list
Package     Version
----------- -------
pip         23.3.2
setuptools  69.0.3
version_pkg 0.1
wheel       0.42.0
ichard26@asus-ubuntu:~/dev/oss/pip/temp/pip-test-package$ ls -gha ~/dev/oss/pip/venv/lib/python3.11/site-packages/ | grep "version"
-rw-rw-r--  1 ichard26   50 Jan  1 16:56 version_pkg.egg-link

[matthuisman]

there is some issue with 69.0.3 and pip and editable installs

Python 3.10.9 Pip 23.3.2

Directory name: sys_testcases Package name in setup.py: sys_testcases pip install -e .

pip list

(its not shown as editable)

If I change the package name to sys-testcases - it works as expected

[abravalheri]

Hi @ichard26, thank you very much for reporting the issue.

If I had to guess, I would say this is probably related to the change in https://github.com/pypa/setuptools/pull/4159, which was motivated by https://github.com/pypa/setuptools/issues/2522.

Probably a previous implementation was optimised to not need a second pass of to_filename, but once safe_name was modified, the circumstances changed and the observed behaviour deviated from the docs (an oversight that never got flagged out and fixed).

We could do an extra pass of to_filename after safe_name for the .egg-link file to be compliant with the doc, but as you pointed out yourself it would not change anything and the filename would still have an underscore in the test case you are analysing... Would that be the acceptable "new normal" for pip? Should we favour instead "behaviour from 69.0.2" over "documentation"?

@jaraco what is your opinion on what is the best approach here?

[ichard26]

We essentially have three options:

1. Update setuptools to maintain pre v69.0.3 naming behaviour -> pip remains unchanged
2. Update setuptools to honour its documentation -> pip still needs an update
3. Only update pip to look for both pre-v69.0.3 and post-v69.0.3 egg-link files -> setuptools remains unchanged

No. 1 would lead to the least fallout and is my preferred option, but I have zero stake in this so I'll defer to setuptools and pip developers to decide on the right path forward.

[sbidoul]

@jaraco @abravalheri is there any chance you can revert to the previous way of naming .egg-link files (assuming this is technically feasible)? This broke pip freeze but also pip uninstall of legacy editable installs. I'm painfully coming up with a pip update but this change is nevertheless quite disruptive and in a legacy area that I'd prefer to leave alone.

[sbidoul]

The pip change is in https://github.com/pypa/pip/pull/12477. The modified egg-link file detection is in src/pip/_internal/utils/egg_link.py.

[ablatner]

My project's constraint autoupdater is affected by this. After bumping setuptools to 69.0.3, subsequent updates with pip list -e don't recognize editable installs with underscores. For now, we can pin setuptools to 69.0.2, but I'm curious what we should do going forwards.

@sbidoul, will the next release of pip include https://github.com/pypa/pip/pull/12477, and will that provide compatibility with setuptools 69.0.3?

[sbidoul]

will the next release of pip include https://github.com/pypa/pip/pull/12477, and will that provide compatibility with setuptools 69.0.3?

@ablatner yes, it is merged so it will be in the next pip release.

[jaraco]

Apologies for being late to the conversation. It's been a rough couple of months for me personally. I got almost no open source work done during the winter break (where I usually have my most productive period).

Here is the design principles I'd like to advocate for:

1. Most importantly, valid characters in the user's indicated package name should be retained in the metadata.
2. Any conversion (normalization/mangling) that occurs should be based on a standard and implemented in a re-usable function (like safe_name).
3. The naming on disk should be relatively stable and predictable. Unit tests should protect this expectation.

is there any chance you can revert to the previous way of naming .egg-link files (assuming this is technically feasible)?

Yes, probably, and that would honor (3). I think that's what @abravalheri was suggesting.

It was unintentional that the name of the metadata path on disk changed. I'm slightly surprised that the test suite didn't manifest failures for this missed expectation.

In #4159, I did propose that the behavior could be rolled back if it caused unexpected disruption, which it did, to be followed by a more involved approach. That's still an option.

@sbidoul Given the work on pip, what would you like to see from setuptools at this point?

[sbidoul]

Given the work on pip, what would you like to see from setuptools at this point?

@jaraco The compatibility work is done in pip indeed. Independently of that, to minimize issues for user who don't have the (yet unreleased) latest pip, I think it would be preferable to preserve the .egg-link file naming (the .egg-info naming changing does not seem to cause issues although I have not looked closely).

[jaraco]

@mattip had this to say in https://github.com/pypa/setuptools/pull/4159#issuecomment-1928850518.

I think this might have changed wheel building for projects with a - in the project name (from the project's pyproject.toml). Now the wheel filename for a project like scipy-openblas64 will become scipy_openblas64-0.3.26-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl. When using twine to upload to PyPI the name is converted back to scipy-openblas64, but the anaconda uploader will respect the _ and try to upload to scipy_openblas64. See scientific-python/upload-nightly-action#61

[jaraco]

Now I'm beginning to wonder if the symptom reported by mattip is in fact a completely different issue, as it deals with the name of the wheel (not the egg-link). Therefore, I'm going to explore that concern separately in #4214.

[jaraco]

I've thusfar been unsuccessful in reproducing the issue.

``` draft @ py -m venv .venv draft @ with open("setup.py", "w") as f: ........ f.write("""\ ........ from setuptools import setup, find_packages ........ setup( ........ name="version_pkg", ........ version="0.1", ........ packages=find_packages(), ........ py_modules=["version_pkg"], ........ )""") ........ ........ with open("version_pkg.py", "w") as f: ........ f.write("""\ ........ def main(): ........ print('0.1')""") ........ 157 28 draft @ py -m pip install -e . Obtaining file:///Users/jaraco/draft Installing build dependencies ... done Checking if build backend supports build_editable ... done Getting requirements to build editable ... done Preparing editable metadata (pyproject.toml) ... done Building wheels for collected packages: version_pkg Building editable for version_pkg (pyproject.toml) ... done Created wheel for version_pkg: filename=version_pkg-0.1-0.editable-py3-none-any.whl size=2472 sha256=0ba192e11f124648598d771979b50868536f3c8a8156edb745a02a55cd2589cc Stored in directory: /private/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-ephem-wheel-cache-jywgfrm3/wheels/af/49/6b/4feeb3d319be8bf02986d3c092b59a17ae5a88757cbf49cf5b Successfully built version_pkg Installing collected packages: version_pkg Successfully installed version_pkg-0.1 [notice] A new release of pip is available: 23.3.1 -> 24.0 [notice] To update, run: /Users/jaraco/draft/.venv/bin/python -m pip install --upgrade pip draft @ py -m pip list Package Version Editable project location ----------- ------- ------------------------- pip 23.3.1 version_pkg 0.1 /Users/jaraco/draft [notice] A new release of pip is available: 23.3.1 -> 24.0 [notice] To update, run: /Users/jaraco/draft/.venv/bin/python -m pip install --upgrade pip draft @ py -m pip freeze # Editable install with no version control (version_pkg==0.1) -e /Users/jaraco/draft draft @ py -m pip uninstall version-pkg Found existing installation: version_pkg 0.1 Uninstalling version_pkg-0.1: Would remove: /Users/jaraco/draft/.venv/lib/python3.12/site-packages/__editable__.version_pkg-0.1.pth /Users/jaraco/draft/.venv/lib/python3.12/site-packages/__editable___version_pkg_0_1_finder.py /Users/jaraco/draft/.venv/lib/python3.12/site-packages/version_pkg-0.1.dist-info/* Proceed (Y/n)? y Successfully uninstalled version_pkg-0.1 draft @ py -m pip install -U setuptools Collecting setuptools Using cached setuptools-69.0.3-py3-none-any.whl.metadata (6.3 kB) Using cached setuptools-69.0.3-py3-none-any.whl (819 kB) Installing collected packages: setuptools Successfully installed setuptools-69.0.3 [notice] A new release of pip is available: 23.3.1 -> 24.0 [notice] To update, run: /Users/jaraco/draft/.venv/bin/python -m pip install --upgrade pip draft @ py -m pip install -e . Obtaining file:///Users/jaraco/draft Installing build dependencies ... done Checking if build backend supports build_editable ... done Getting requirements to build editable ... done Preparing editable metadata (pyproject.toml) ... done Building wheels for collected packages: version_pkg Building editable for version_pkg (pyproject.toml) ... done Created wheel for version_pkg: filename=version_pkg-0.1-0.editable-py3-none-any.whl size=2472 sha256=bbecd6be494c2a4f485ee20fbabb4167219935c97d220a8416d1f7ab68a29962 Stored in directory: /private/var/folders/f2/2plv6q2n7l932m2x004jlw340000gn/T/pip-ephem-wheel-cache-opni86bz/wheels/af/49/6b/4feeb3d319be8bf02986d3c092b59a17ae5a88757cbf49cf5b Successfully built version_pkg Installing collected packages: version_pkg Successfully installed version_pkg-0.1 [notice] A new release of pip is available: 23.3.1 -> 24.0 [notice] To update, run: /Users/jaraco/draft/.venv/bin/python -m pip install --upgrade pip draft @ py -m pip freeze setuptools==69.0.3 # Editable install with no version control (version_pkg==0.1) -e /Users/jaraco/draft draft @ py -m pip list Package Version Editable project location ----------- ------- ------------------------- pip 23.3.1 setuptools 69.0.3 version_pkg 0.1 /Users/jaraco/draft [notice] A new release of pip is available: 23.3.1 -> 24.0 [notice] To update, run: /Users/jaraco/draft/.venv/bin/python -m pip install --upgrade pip ```

What am I missing to trigger the behavior?

[sbidoul]

In your example above pip does a PEP 660 install, so no egg-link is involved. Installing wheel in the venv should cause pip to do setup.py develop.

[jaraco]

Ah, yes. Confirmed. And also confirmed that pip<24 is now required as well to replicate the issue:

``` i4167 @ py -m pip install wheel Collecting wheel Using cached wheel-0.42.0-py3-none-any.whl.metadata (2.2 kB) Using cached wheel-0.42.0-py3-none-any.whl (65 kB) Installing collected packages: wheel Successfully installed wheel-0.42.0 [notice] A new release of pip is available: 23.3.2 -> 24.0 [notice] To update, run: /Users/jaraco/draft/i4167/.venv/bin/python -m pip install --upgrade pip i4167 @ py -m pip uninstall version-pkg Found existing installation: version_pkg 0.1 Uninstalling version_pkg-0.1: Would remove: /Users/jaraco/draft/i4167/.venv/lib/python3.11/site-packages/__editable__.version_pkg-0.1.pth /Users/jaraco/draft/i4167/.venv/lib/python3.11/site-packages/__editable___version_pkg_0_1_finder.py /Users/jaraco/draft/i4167/.venv/lib/python3.11/site-packages/version_pkg-0.1.dist-info/* Proceed (Y/n)? y Successfully uninstalled version_pkg-0.1 i4167 @ py -m pip install -e . Obtaining file:///Users/jaraco/draft/i4167 Preparing metadata (setup.py) ... done Installing collected packages: version_pkg Running setup.py develop for version_pkg Successfully installed version_pkg-0.1 [notice] A new release of pip is available: 23.3.2 -> 24.0 [notice] To update, run: /Users/jaraco/draft/i4167/.venv/bin/python -m pip install --upgrade pip i4167 @ py -m pip list Package Version ----------- ------- pip 23.3.2 setuptools 69.0.3 version_pkg 0.1 wheel 0.42.0 [notice] A new release of pip is available: 23.3.2 -> 24.0 [notice] To update, run: /Users/jaraco/draft/i4167/.venv/bin/python -m pip install --upgrade pip i4167 @ py -m pip install -U pip Requirement already satisfied: pip in ./.venv/lib/python3.11/site-packages (23.3.2) Collecting pip Using cached pip-24.0-py3-none-any.whl.metadata (3.6 kB) Using cached pip-24.0-py3-none-any.whl (2.1 MB) Installing collected packages: pip Attempting uninstall: pip Found existing installation: pip 23.3.2 Uninstalling pip-23.3.2: Successfully uninstalled pip-23.3.2 Successfully installed pip-24.0 i4167 @ py -m pip list Package Version Editable project location ----------- ------- ------------------------- pip 24.0 setuptools 69.0.3 version_pkg 0.1 /Users/jaraco/draft/i4167 wheel 0.42.0 ```

[jaraco]

I've tested with both version_pkg and version-pkg as the name and can confirm what's been stated above.

1. Setuptools is not doing what the docs say. The docs say the filename should be "named following the format for .egg and .egg-info". That is, it should be "version_pkg.egg-link" in both cases (to_filename applied). I presume Setuptools never did honored the specified naming and so the implementation has always been out-of-spec for packages named with a dash.
2. Setuptools docs are wrong when they say "These latter two functions (safe_name and safe_version) can also be used to later 'unescape' these parts of the filename." You can of course convert underscores back to dashes, but you'll never know if the original character in the name was an underscore or a dash. That sentence should just be deleted or maybe be replaced by an acknowledgement that these transformations are irreversible.
3. Prior to Setuptools 69.0.3, the implementation was additionally broken in that the egg-link names did not get the to_filename transformation, so packages named with an underscore were incorrectly transformed into a dash-separated name (against the docs), conflicting with the convention of other filenames.

Here's what I propose:

• [ ] Restore the prior behavior by wrapping the egg-link name generation in a _to_filename_broken (or similar), documenting the fact that this transformation is incorrect but needed to maintain compatibility with pip<24.
• [ ] Since pip has graciously fixed the implementation to honor both - and _ separators, give a generous compatibility period of 6-24 months, but then replace _to_filename_broken with to_filename (or similar), honoring the docs and depending on pip 24 or later.
• [ ] Update the docs per (2) above.
• [ ] Also, in the docs, point to this issue, and
• [ ] Ultimately point to the minimum setuptools version required to honor the docs.

[ablatner]

I had pinned my project to setuptools 69.0.2 until pip was updated to 24.0. I updated setuptools to 69.1.0, and the next constraint update broke my editable installs in the same way as before.

[ablatner]

@jaraco @sbidoul do you have any thoughts?

[sbidoul]

do you have any thoughts?

@ablatner can you provide a reproducer?

[ablatner]

Unfortunately it is in a private corporate repository. Maybe to take a step back, should we expect a difference in package names for the command python3 -m pip list -e, between these configurations?

1. setuptools=69.0.2, pip=23.2
2. setuptools=69.1.0, pip=24.0

[chrisjbillington]

This seems related: in setuptools 69.5.1, I'm seeing that for a package called my-package, Command.distribution.get_name() returns my-package, but Command.distribution.get_fullname() returns my_package-<version>. So that seems inconsistent.

I was using f'{self.distribution.get_name()}-{self.VERSION}.tar.gz' to get the name of the resulting sdist, which now is incorrect. I am now switching to f'{self.distribution.get_fullname()}.tar.gz', which works for now, if it's likely to stop working I'd be interested to know!

(.tar.gz is guaranteed since I pass --formats=gztar).