Open ichard26 opened 9 months ago
there is some issue with 69.0.3 and pip and editable installs
Python 3.10.9 Pip 23.3.2
Directory name: sys_testcases Package name in setup.py: sys_testcases pip install -e .
pip list
(its not shown as editable)
If I change the package name to sys-testcases - it works as expected
Hi @ichard26, thank you very much for reporting the issue.
If I had to guess, I would say this is probably related to the change in https://github.com/pypa/setuptools/pull/4159, which was motivated by https://github.com/pypa/setuptools/issues/2522.
Probably a previous implementation was optimised to not need a second pass of to_filename
, but once safe_name
was modified, the circumstances changed and the observed behaviour deviated from the docs (an oversight that never got flagged out and fixed).
We could do an extra pass of to_filename
after safe_name
for the .egg-link
file to be compliant with the doc, but as you pointed out yourself it would not change anything and the filename would still have an underscore in the test case you are analysing... Would that be the acceptable "new normal" for pip
? Should we favour instead "behaviour from 69.0.2" over "documentation"?
@jaraco what is your opinion on what is the best approach here?
We essentially have three options:
No. 1 would lead to the least fallout and is my preferred option, but I have zero stake in this so I'll defer to setuptools and pip developers to decide on the right path forward.
@jaraco @abravalheri is there any chance you can revert to the previous way of naming .egg-link
files (assuming this is technically feasible)? This broke pip freeze
but also pip uninstall
of legacy editable installs. I'm painfully coming up with a pip update but this change is nevertheless quite disruptive and in a legacy area that I'd prefer to leave alone.
The pip change is in https://github.com/pypa/pip/pull/12477. The modified egg-link file detection is in src/pip/_internal/utils/egg_link.py.
My project's constraint autoupdater is affected by this. After bumping setuptools to 69.0.3, subsequent updates with pip list -e
don't recognize editable installs with underscores. For now, we can pin setuptools to 69.0.2, but I'm curious what we should do going forwards.
@sbidoul, will the next release of pip include https://github.com/pypa/pip/pull/12477, and will that provide compatibility with setuptools 69.0.3?
will the next release of pip include https://github.com/pypa/pip/pull/12477, and will that provide compatibility with setuptools 69.0.3?
@ablatner yes, it is merged so it will be in the next pip release.
Apologies for being late to the conversation. It's been a rough couple of months for me personally. I got almost no open source work done during the winter break (where I usually have my most productive period).
Here is the design principles I'd like to advocate for:
safe_name
).is there any chance you can revert to the previous way of naming
.egg-link
files (assuming this is technically feasible)?
Yes, probably, and that would honor (3). I think that's what @abravalheri was suggesting.
It was unintentional that the name of the metadata path on disk changed. I'm slightly surprised that the test suite didn't manifest failures for this missed expectation.
In #4159, I did propose that the behavior could be rolled back if it caused unexpected disruption, which it did, to be followed by a more involved approach. That's still an option.
@sbidoul Given the work on pip, what would you like to see from setuptools at this point?
Given the work on pip, what would you like to see from setuptools at this point?
@jaraco The compatibility work is done in pip indeed. Independently of that, to minimize issues for user who don't have the (yet unreleased) latest pip, I think it would be preferable to preserve the .egg-link
file naming (the .egg-info
naming changing does not seem to cause issues although I have not looked closely).
@mattip had this to say in https://github.com/pypa/setuptools/pull/4159#issuecomment-1928850518.
I think this might have changed wheel building for projects with a
-
in the project name (from the project'spyproject.toml
). Now the wheel filename for a project likescipy-openblas64
will becomescipy_openblas64-0.3.26-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
. When usingtwine
to upload to PyPI the name is converted back toscipy-openblas64
, but the anaconda uploader will respect the_
and try to upload toscipy_openblas64
. See scientific-python/upload-nightly-action#61
Now I'm beginning to wonder if the symptom reported by mattip is in fact a completely different issue, as it deals with the name of the wheel (not the egg-link). Therefore, I'm going to explore that concern separately in #4214.
I've thusfar been unsuccessful in reproducing the issue.
What am I missing to trigger the behavior?
In your example above pip does a PEP 660 install, so no egg-link is involved.
Installing wheel
in the venv should cause pip to do setup.py develop
.
Ah, yes. Confirmed. And also confirmed that pip<24
is now required as well to replicate the issue:
I've tested with both version_pkg
and version-pkg
as the name and can confirm what's been stated above.
.egg
and .egg-info
". That is, it should be "version_pkg.egg-link" in both cases (to_filename
applied). I presume Setuptools never did honored the specified naming and so the implementation has always been out-of-spec for packages named with a dash.safe_name
and safe_version
) can also be used to later 'unescape' these parts of the filename." You can of course convert underscores back to dashes, but you'll never know if the original character in the name was an underscore or a dash. That sentence should just be deleted or maybe be replaced by an acknowledgement that these transformations are irreversible.to_filename
transformation, so packages named with an underscore were incorrectly transformed into a dash-separated name (against the docs), conflicting with the convention of other filenames.Here's what I propose:
_to_filename_broken
(or similar), documenting the fact that this transformation is incorrect but needed to maintain compatibility with pip<24
.-
and _
separators, give a generous compatibility period of 6-24 months, but then replace _to_filename_broken
with to_filename
(or similar), honoring the docs and depending on pip 24 or later.I had pinned my project to setuptools 69.0.2 until pip was updated to 24.0. I updated setuptools to 69.1.0, and the next constraint update broke my editable installs in the same way as before.
@jaraco @sbidoul do you have any thoughts?
do you have any thoughts?
@ablatner can you provide a reproducer?
Unfortunately it is in a private corporate repository. Maybe to take a step back, should we expect a difference in package names for the command python3 -m pip list -e
, between these configurations?
This seems related: in setuptools 69.5.1, I'm seeing that for a package called my-package
, Command.distribution.get_name()
returns my-package
, but Command.distribution.get_fullname()
returns my_package-<version>
. So that seems inconsistent.
I was using f'{self.distribution.get_name()}-{self.VERSION}.tar.gz'
to get the name of the resulting sdist, which now is incorrect. I am now switching to f'{self.distribution.get_fullname()}.tar.gz'
, which works for now, if it's likely to stop working I'd be interested to know!
(.tar.gz
is guaranteed since I pass --formats=gztar
).
setuptools version
69.0.3
Python version
CPython 3.11.7
OS
Ubuntu 22.04.03 LTS
Additional environment information
No response
Description
Happy New Year! :tada:
I was debugging pip's CI which has been red ever since the release of Setuptools 69.0.3. Most of them are easily fixed by removing the underscore -> dash normalization assumption, however, there is one place where leaving the underscores intact causes issues: legacy editable installs.
As [documented in the Setuptools documentation](https://setuptools.pypa.io/en/latest/deprecated/python_eggs.html#:~:text=The%20%E2%80%9Cname%E2%80%9D%20and%20%E2%80%9Cversion%E2%80%9D%20should%20be%20escaped%20using%20the%20to_filename()%20function%20provided%20by%20pkg_resources%2C%20after%20first%20processing%20them%20with%20safe_name()%20and%20safe_version()%20respectively.), the distribution name part of egg filenames should be normalized by
safe_name()
:Setuptools does not honour this.[^1] This is actually fine in most situations as far as I can tell since there are modern ways for pip to discover installed distributions that don't rely on eggs, but
setup.py develop
does not generate this modern metadata. Thus, pip falls back to searchingsys.path
for a.egg-link
file to determine whether a distribution is editably installed. Pip assumes the egg link name will be normalized bysafe_name()
so this logic returns a false negative despiteversion_pkg
being editably installed in fact.If I'm being honest, I have no idea whose problem this is, but this does mean for projects that do not implement PEP 518 and have underscores in their name will not be recognized as editably installed. If this is better transferred to the pip repository, please let me know!
[^1]: Well, if you read the documentation closely, apparently you're supposed to pass the made safe name to
pkg_resources.to_filename()
which would turn the dash back into an underscore but setuptools does not seem to do this anyway /shrugExpected behavior
See above.
How to Reproduce
pip<24
,setuptools==69.0.3
andwheel
.version_pkg
:pip install -e .
in the directorypip freeze
orpip list
and observe that pip doesn't realize it's an editable installversion_pkg
is installed as an editableOutput