Open webknjaz opened 3 years ago
I'd like to voice my -1 -- universal just means "pure python and works on any version" and presumably will hold the same meaning when C-incompatible python 4.x rolls around. For the same reason that python_requires = <4
is bad I don't think this should be implemented.
@asottile that's what I was thinking until @gaborbernat pointed out that it's not exactly true. It is currently py2.py3
in the wheel names. How do you suggest we should lint the conflicting metadata?
It is true, or at at least what the documented point of --universal
was: https://github.com/pypa/wheel/commit/4a1b292b73a20e71f3939decbbb771507da79ce7#diff-8d068e8797e88947c320f79e856c3e16a72b730124a8f9d7031e2c4680dfa534R110-R111
the metadata isn't conflicting, pip
properly handles python_requires
and will only consider the wheel on appropriate python 3 versions
equates to the tag "py2.py3".
This is hinting pip to attempt downloading the wheel under Python 2. @pradyunsg seems to think that this could cause problems with the old resolver (combined with Requires-Python: >=3.6
, for example) because it doesn't know how to do backtracking.
Also, I think that it's not the config for the tool producing wheels that should be linted but the resulting metadata in the wheel.
I'd like to voice my -1 -- universal just means "pure python and works on any version" and presumably will hold the same meaning when C-incompatible python 4.x rolls around. For the same reason that
python_requires = <4
is bad I don't think this should be implemented.
To achieve that intent the wheel tag this should map would need to be just py
, not py2.py3
. because py2.py3
is not python 4 compatible tag.
It would only cause problems for pip versions that don't understand python_requires
(<9) -- which at this point they have much much harder problems installing valid packages
what I'd rather see is if python2 is truly dead then wheel
/ pip
/ setuptools
should stop producing py2
in the name instead of requiring me to remove valid metadata from my configuration (which I use to signal that my package is pure python) I'll have to annoyingly add back later
To achieve that intent the wheel tag this should map would need to be just
py
, notpy2.py3
. becausepy2.py3
is not python 4 compatible tag.
I like this idea! let's push to make that happen
I mean, either universal should start meaning py
or python 3+
packages should stop using it. The current universal translation to py2.py3
means generated packages end up in an undesirable state because they are incompatible with python 2. At the same time, they contain code that works only with Python 3+.
Can we get some opinion from at least:
Thanks!
I think https://github.com/pypa/wheel/issues/396 is on the right track, except that I would rather abort the operation if such a discrepancy is detected because otherwise this could go unnoticed by the users.
As packaging BDFL, the wheel spec doesn't mention "universal" at all, so I'd consider that an implementation detail of the wheel package.
From pip's POV, we consume the tags, not "universal" directly. I'd be happy if pip were to add a check that the tags and Python-Requires
were consistent, but I'd see that as a quality of implementation matter. For correctness, what matters is that pip only considers wheels with supported tags, and it rejects any where Python-Requires
isn't satisfied. That's well-defined behaviour even if the tags and python-requires conflict.
Yep, universal
is an implementation detail in the wheel
package. This feature will be removed in the eventual 1.0.0 release. In the meantime, raising an error when a conflict is detected between python_requires
and universal
is my current game plan.
Yep,
universal
is an implementation detail in thewheel
package. This feature will be removed in the eventual 1.0.0 release. In the meantime, raising an error when a conflict is detected betweenpython_requires
anduniversal
is my current game plan.
This will break a lot of CIs, rightfully though.
This will break a lot of CIs, rightfully though.
As a precaution we could check some of the most popular packages to make sure they have their settings right. I could probably write a script for this.
What information are you looking out for here? I mean, almost all of @asottile packages will fail essentially, and I'm sure even I have a few lying around (though I don't mind if the CI starts failing). You could in theory get https://hugovk.github.io/top-pypi-packages/, and download each wheel, check if the wheel tag is conflicting with python-reqires. But say 60% fail, what would you do then?
Pre-built wheels are not a problem, since wheel
won't touch those. The problem lies with those who, for whatever reason, do their builds from source packages.
I would lessen the impact by submitting PRs to projects that have misconfigured their packaging setup.
I should also say that right now, there's nothing in any spec that disallows publishing a wheel with (for example) a Python tag of .py37
and a Python-Requires
of >=3.8
. Such a wheel is uninstallable, but it's not invalid. I see no reason to object if someone were to propose a spec change that tightened this up, but there isn't anything right now, so it's arguably not standards-compliant to reject such wheels. But it seems like a sensible thing to do, if you want to. It's certainly not mandated that you do so, but I'd have little sympathy for anyone who demanded that tools allow uninstallable wheels to be published ๐
Wheels where the tags and requires-python describe different but overlapping sets of versions is a much harder problem, though (for a start, there's no set of tags that's equivalent to an open-ended requires-python like >= 3.7
). I'd say that it's unlikely to be worth doing anything on that in a standards context. But if someone wants to try, I'm happy to listen to the debate and see where it goes ๐
And that's about all that I think needs to be said from a standards perspective.
To achieve that intent the wheel tag this should map would need to be just
py
That would mean you'd need a brand new pip version (and brand new packaging version), which would never work on Python 2 since that's been dropped by both packages. Not against a "py" tag, but it's come too late for this. If it was implemented, it probably would need a new setting; [bdist_wheel] universal=1
is pretty arcane; why do you have to tell bdist_wheel
something for a normal, pure Python package?
universal just means "pure python and works on any version"
It's very clearly documented that this means Python 2 + Python 3, from the first release note (wheel 0.10, "Define a new setup.cfg section [wheel]. universal=1 will apply the py2.py3-none-any tag for pure python wheels."), two years before the change linked above (which also still mentions py2.py3
). It never was integrated with the Requires-Python, so it always meant "py2.py3" because that's what "universal" meant. Wheel already can detect pure-python; as far as I know, it always has (no Extension present); there's no reason to duplicate this information. It has been clearly documented that a Pure Python wheel is a pure Python wheel that does not support Python 2.
almost all of @asottile packages will fail essentially
He can change one package (setup-cfg-fmt
) and they all will be instantly fixed, I expect ;) . And that will help clean up mine too if I have any laying around. :) (And I did have to fix a package about a month ago, so it does happen). That's one reason I really wanted to convince him that this should not be set. I miserably failed, he said it's about pure Python, not about Python 2.
I would lessen the impact by submitting PRs to projects that have misconfigured their packaging setup.
I did. Didn't get a happy response. https://github.com/pre-commit/pre-commit/pull/1832. "general word of advice: don't touch infrastructure or tooling for a project you don't maintain unless prompted by an issue asking explicitly to do so" and "the crusade to remove this flag is not correct and I closed this change because it is not welcome here".
a Python tag of .py37 and a Python-Requires of >=3.8. Such a wheel is uninstallable,
technically that could be installable since py37-none-any
means purelib 3.7+ -- if instead that's cp37-cp37m
then yes it is uninstallable
All older (not c) python3 tags install:
python3.9 -m pip debug -v
...
Compatible tags: 1776
...
py30-none-macosx_10_4_universal
cp39-none-any
py39-none-any
py3-none-any
py38-none-any
py37-none-any
py36-none-any
py35-none-any
py34-none-any
py33-none-any
py32-none-any
py31-none-any
py30-none-any
I would also like to deprecate the universal
setting already, but what would the proper replacement be? Should wheels be universal by default if there is no python_requires
? Probably not. Perhaps then a python_requires
that includes Python 2.x?
There's an idea to set the lowest Python supported as the tag via Requires-Python here: https://github.com/pypa/wheel/issues/336 , by the way.
I would also like to deprecate the
universal
setting already, but what would the proper replacement be?
I don't see a strong point for producing py2.py3
. It was a very special situation that hopefully won't be repeated, and most packages are now py3
. I would actually rather hope py
does get added and turned on in 3-5 years so we don't end up with py3.py4
if there's a Python 4; this should be handled by Requires-Python in the future, rather than tags.
I'm not against getting more packages using the python_requires
setting. It's surprisingly rare in cibuildwheel's known users, about 50%.
technically that could be installable since py37-none-any means purelib 3.7+ -- if instead that's cp37-cp37m then yes it is uninstallable
Drat, that'll teach me to try to be clever and invent a "better" example (I was originally going with py2 rather than py37, which would also be uninstallable, I believe - although now I'm nervous even about that ๐) And actually, there's no specification that mandates what compatibility tags a given distribution must say it supports, so from a purely theoretical standpoint, it's even more complex than that...
Let's just go with "it's possible to create wheels that are uninstallable" ๐
@asottile Do you still object to the removal of universal=1
in your projects?
@asottile Do you still object to the removal of
universal=1
in your projects?
correct
Why? When it's clearly wrong to have this in a py3 only project?
I don't see why a user should be expected to make or understand some random universal=1
call to bdist_wheel
in setup.cfg
. It doesn't translate to PEP 621 configuration, and it is and was never needed to indicate "this is pure python" - the absence of Extensions does that. It was intended from day one (wheel 0.10.0 in 2012, when it was called [wheel] universal=1
) to provide a way to go against the original design that Python 2 and Python 3 code could not be run without changes (read the Python 3.0 changelog for the sad statements like that). If we want a "pure_python = true" flag, then it could be added as such, and not as a misused bdist_wheel
option; but it's not needed, we don't need a flag that says this is pure Python, and we already have a lot of cruft to just make a simple package (see flit and poetry for attempts to make things shorter/simpler).
For Python 4, we have Requires-Python and the knowledge that people are going to want to write code that runs in both Python 3 and 4 at the same time.
From the 3.0 release notes:
It is not recommended to try to write source code that runs unchanged under both Python 2.6 and 3.0; youโd have to use a very contorted coding style, e.g. avoiding print statements, metaclasses, and much more. If you are maintaining a library that needs to support both Python 2.6 and Python 3.0, the best approach is to modify step 3 above by editing the 2.6 version of the source code and running the 2to3 translator again, rather than editing the 3.0 version of the source code.
The universal=True
option was a stopgap measure which was introduced long before requires_python
became a thing.
bdist_wheel
is also becoming an implementation detail, and not something users are expected to call by hand; it's called by pypa/pip and now by pypa/build. Having it in the config seems counter productive to that work.
Yes, and setuptools is likely to adopt the bdist_wheel command into itself in the future.
It was intended from day one (wheel 0.10.0 in 2012, when it was called [wheel] universal=1) to provide a way to go against the original design that Python 2 and Python 3 code could not be run without changes (read the Python 3.0 changelog for the sad statements like that).
Do you have any documentation that confirms that's how "universal" was intended? I don't recall it being like that - it was originally intended specifically to indicate that a project was "universal" as in "could be run on any Python installation" and as such translated to py2.py3
precisely because that was "any Python installation" at the time. It wasn't "going against the design", although it probably was acknowledging that people were writing cross-version code even though that wasn't how the core devs imagined the transition would work.
The universal=True option was a stopgap measure which was introduced long before requires_python became a thing.
Um, requires_python
defines a limited range of Python versions, whereas "universal" says "anything works" (ignoring py4
...). They are pretty much opposites of each other, so I'm not sure how universal could be a stopgap until requires_python
came along.
It's certainly true that things have changed, though, and the reality nowadays is that "universal" probably isn't a useful designation for real projects, and certainly there are unlikely to be many projects that still work on "all versions of Python". But I'd suggest that we stick to discussion what's useful or worth supporting now, and not try to read too much into the history (or people's recollections of it).
Um, requires_python defines a limited range of Python versions, whereas "universal" says "anything works" (ignoring py4...). They are pretty much opposites of each other, so I'm not sure how universal could be a stopgap until requires_python came along.
Well, as you pointed out, it scopes the wheels to Python 2 and 3 only which is not really universal. But realistically, any project has a minimum supported Python version which they should indicate with requires_python
. In my opinion this should be a requirement for any packages uploaded to PyPI.
it scopes the wheels to Python 2 and 3 only which is not really universal.
It was when we designed the flag. We didn't think sufficiently far ahead, but the intention was "everything".
But realistically, any project has a minimum supported Python version which they should indicate with requires_python
But what I support and what it works on may be two different things. And I may prefer not to block people on older systems from using my code, just because I won't support them. I feel like there's an assumption underlying this that all projects are somehow required to hit a certain bar of "professionalism" here, and to be blunt, that just isn't the case.
In my opinion this should be a requirement for any packages uploaded to PyPI.
How do you plan on fixing all the existing projects that don't do that? I really don't see any benefit in a rule like this that only applies to new uploads.
To give a concrete example, pip's resolver, if faced with the latest version of a package saying that it doesn't support my Python version, will quite happily go back through all the older versions (potentially hundreds of them) looking for one that supports my installation. And ultimately, what it will find won't necessarily be one that supports my version, it's just as likely to find something that's old enough that it was uploaded before the rule requiring requires_python
was put in place, and give me some ancient junk that still doesn't work on my system.
We need to remember that PyPI don't have the resource (or, as far as I'm aware, the desire) to be a curated repository of packages, so there's a lot of stuff on there that's of questionable value or is simply abandoned. Imposing new rules without considering how we apply them to existing projects is just going to result in rules that have so many exceptions that people of necessity will ignore them.
Also, what's to stop projects just saying requires_python >1.0
? How does mandating the field is present make any practical difference?
Anyway, this is going a long way off topic. I'm going to stop responding to this issue unless explicitly pinged, as I don't think this subthread is going anywhere useful.
I suggest we get back to the original question, of whether twine check
should let people know if their python_requires
and their wheel tags are inconsistent, and leave the wider infrastructure questions for another time.
Do you have any documentation that confirms that's how "universal" was intended?
The guess about intentions are based on the problem it was trying to solve (memory, the current docs, and old release notes); I just downloaded wheel 0.9.7 and setuptools 0.7.2, and ran bdist_wheel on a simple project. The produced wheel was example_pkg_YOUR_USERNAME_HERE-0.0.1-py27-none-any.whl
- this would load on a hypothetical Python 2.8, but not on Python 3. The universal
indicated that it would load on Python 3 as well. It also had the nice side effect of allowing it to load on older versions of Python (2.6, for example), but that's not the case anymore. So there was a period between 2012 and 2014 when this also had a nice side effect for Pure Python wheels. This was fixed with wheel 0.23.0 in March, 2014; the default tag became py2
or py3
for pure Python projects. The "universal" flag has only meant "supports Python 2 and 3" since then.
Note that @asottile's projects are Python 3.6+, and have requires_python= >=3.6.1
, but he wants to keep the universal=1
and not support Python 2 since: (@asottile please correct me if wrong, based on a message you sent earlier, and I don't really agree so I'm sure I'm not phrasing it well)
For 1, since setuptools, pip, packaging, and most other Python 3+ pure Python libraries have dropped it, I don't think that's true anymore. It's also very clearly documented everywhere it shows up in wheel and packaging.python.org's guides that it means Python 2 + 3, and pure Python wheels do not need universal=1.
For Python 4, I don't think we could reuse [bdist_wheel] universal=1
. It clearly currently means py2.py3
, and will there really be a py2.py3.py4
? Where does it end? That's why Requires-Python is better; every package with any code in it to speak of has some minimum Python version - it's not Universal. Also, the meaning of Universal should change over time, but packaging shouldn't change over time. Even 2.7, 3.5+ isn't really "universal", which is also why python_requires=...
is better. So it's not really currently useful, and it's probably not useful in the future.
For macOS, "Universal" meant PowerPC+Intel. "Fat" meant Intel32+Intel64. And now "Universal2" means Intel+AppleSilicon. I think we could avoid another special flag to bdist_wheel by using Requires-Python, but say we didn't; it probably would have to be universal2=True
or just the tag, py=py3.py4
, etc. Actually, py36.py41
or something like that would likely be even better and more readable for users. That's what https://github.com/pypa/wheel/issues/336 wants to do.
Note, I do know a few projects that "secretly" supported a version but set the requires one higher. It's not really that helpful of a practice, I think, but it is a rare case where they want to have a py2.py3
tag, since it does work on Python 2, so in a pinch it can be used (though pip won't resolve it, so it's not that useful). But this is a "special" case (and not @asottile's case at all). If it's unofficial support and requires a workaround to install anyway, changing the Python tag isn't that hard.
How about this concrete proposal:
[bdist_wheel] universal=1
: produce py3
wheels, as now.[bdist_wheel] universal=1
: produce py2.py3
wheels, as now. No warning, fully allowed for now.[bdist_wheel] universal=1
: Use the smallest allowed tag for each valid Python major version.
>=2.7
would produce py27.py3
.>=2.6, !=3.0.*, !=3.1.*
would produce py26.py32
.>=3.6
would produce py36
.[bdist_wheel] universal=1
: produce a warning from wheel, maybe eventually an error (after Python 2 is dropped from wheel). This is logically over constrained - it shouldn't universally work on all Pythons and be limited to some Pythons. When producing a warning, it still produces py2.py3
wheels for backward compatibility.This would mostly be implemented through pypa/wheel#336. This has a small consequences to the point of this thread:
Edit: added missing behavior when both are given.
imo attempting to match a wheel tag to python_requires is kind of a lost cause, what would you do for python_requires >= 3.6.1
? (py36
isn't quite right, and py37
definitely isn't correct, and py3
has the same "problems" that py2.py3
has)
Three possibilities.
a) use 99 as the patch; patches can't go above about 18. This worked on all cibuildwheel's users, and has been shipped and has been working for a while (cibuildwheel 1.9 IIRC). This only fails if an upper patch bound is made, which is pretty invalid. This is also clearly suggested in the issue linked above if you read it.
b) Check against a list of 20 patch versions for each Python, and see if they match. If any match, that once can be used. A bit slower.
c) Compute the cross section analytically; if any X.Y.*
is allowed, then use that XY
. More trouble, but probably doable, and should be as fast as a. A SpecifierSet is pretty simple.
I'm also personally fine with py3
, this was based partially on the linked issue wanting to make this more descriptive; there's nothing wrong with a py3 wheel that works on py36+ IMO. py3
means it works on some versions of Python 3; it almost never means it works on 3.0/3.1/3.2. But a py2.py3
file that does not support any version of Python 2 is wrong. Also, having Tag: py2-none-any
in .dist-info/WHEEL
is wrong too.
Similarly, py36
should work on some version of Python 3.6. I'd maybe even state it must work on future versions of Python 3.6 (well, if there were any).
CERN still has running Scientific Linux 6 machines that have pip 8, FFI. It's not truly dead; I'm very glad at least pip correctly removed py2 when updating to 21. 7% of Python 2 users have pip < 9. <unrelated rant>
Though nothing like Pip 9. Ubuntu 18.04 comes with Pip 9! Even on Python 3.8! CentOS 8 comes with Pip 9! Pip 9 can't load manylinux2010+! </unrelated rant>
These sorts of discussions really should be made on the wheel issue, not here.
IMO twine printing warning when there's some sort of obvious mismatch like the valid-but-uninstallable wheel example that @pfmoore gave above, or a wheel that's tagged for Python 2 but can't be installed there due to python_requires is a good idea.
It's probably a good idea to check if the last version of pip that supported Python 2 does not work with py, because if it does, then there's not much discussion to be had here. :)
imo attempting to match a wheel tag to python_requires is kind of a lost cause, what would you do for python_requires >= 3.6.1?
FWIW, I think this issue's OP isn't to suggest that we should try to have them match precisely though. Rather, it's about being tagged for a Python version that the package can't be installed on. And this whole thing about inferring tags should likely move to the wheel issue tracker.
FWIW, I'd like to put the py
tag on this example TBH. Because we don't have that yet, it'd need to be py3
instead (which is fine until we have to figure out what to do for Python 4, and continuing to put py2 here isn't going to help with that problem anyway).
We can add py
in before Python 3/4 though, as @henryiii suggests, but that's probably better for discuss.python.org.
๐ (relatively new) Twine maintainer here. It's been interesting to watch this discussion, but I've only skimmed it. Can somebody write (or point to) a concise proposal for an enhancement to Twine that has (or can get) the support of the folks involved?
@bhrutledge I'd say there does not exist a proposal which everyone who has been involved in this discussion would be supportive of. I'll try to summarize:
py2.py3
because at the time of the implementation that was what "universal" meant (works on 2 and 3)py2.py3
and python-requires
can have > 3.5
I think the best we could do is have a check that can never be considered an error because the confidence in that check would be so low that is also only run when provided a flag that warns if you have requires-python
and the wheel is tagged py2.py3
. That said, we'd be wading into a minefield by doing so and I'm not sure it's worth the effort.
All of the above is to say: There's almost no alignment on what any of this means (the current maintainers of various pypa projects seem to have different interpretations of things compared to the PEPs when they were written) and tools have grown behaviour based on what was available. Twine taking any action here seems like it would result in endless churn and heartburn for us.
Finally as a total aside, it seems all members of PyPA can tag issues on this project despite not being maintainers and thus can give an issue a sense of being accepted as a feature/enhancement. That's kind of crappy and doesn't let the actual maintainers guard their "no" appropriately.
Finally as a total aside, it seems all members of PyPA can tag issues on this project despite not being maintainers and thus can give an issue a sense of being accepted as a feature/enhancement. That's kind of crappy and doesn't let the actual maintainers guard their "no" appropriately.
Oh, sorry about that. I believe that being in the "Gardeners" team grants this right. That team is meant to give other PyPA folks the ability to do some harmless housekeeping but it seems like not everybody's aware of this.
Twine taking any action here seems like it would result in endless churn and heartburn for us.
FWIW I think I'm okay with a "no" if that's what people here think is best. I don't have a strong opinion and only raised this issue because I overlooked this in my projects and didn't even notice until Bernรกt told me.
I think the best we could do is have a check that can never be considered an error because the confidence in that check would be so low that is also only run when provided a flag that warns if you have
requires-python
and the wheel is taggedpy2.py3
.
This solution sounds compelling. I'd definitely +1 it if that's something others can agree on. It is good to have an optionally enabled warning that could become an error with an explicit --strict
(#430).
Pip versions for years have relied on that as a hint as to which wheel should be installed on a given version of Python
To clarify this point slightly, the wheel tags say which Python version(s) a wheel is compatible with, so pip ignores any wheel that has tags saying it isn't compatible with the target system. This is not really a "hint", it's mandated behaviour - you must not install a wheel whose tags declare that it's incompatible.
The only way we use tags to choose which wheel to install, is in the sense that we choose the "most compatible" wheel where multiple otherwise-equivalent wheels are available. "Universal" (whatever that means in practice) will likely be low on that scale, by nature of what it means to be "universal". Again, that's mandated by the standard so all installers will do this, not just pip.
That's kind of crappy and doesn't let the actual maintainers guard their "no" appropriately.
If my comments have had that effect, I'm sorry. I have no authority here and my comments are only intended to be informative. I only got involved because @gaborbernat pinged me for a view on what the standards said, and I've tried my best to confine my comments to clarifications on what the standard-mandated behaviour is (which frankly, is only peripheral, as the issue is about a check, which is not something the standards have anything to say on).
FWIW, I entirely support the project maintainers having the final say, and I do not want any comments I've made here to be seen as exerting any pressure on them.
The discussion here got a little far afield, but the focus is not on universal=1
, which is a custom, non-standards controlled flag directly to the bdist_wheel
command, part of wheel
.
The thing that twine check could (and personally, I think should) to is make sure that the Requires-Python metadata slot and the Python tags (which are both standards controlled) do not conflict. Something with > 3.5
should not have Tag: py2-none-any
. I think that could easily be a warning from twine check. You don't have to follow warnings from twine check.
Whether universal=1
can produce a non-Python 2 tagged wheel is something to take up with wheel, not here. But until something like that is added, Python 3 only wheels from wheel that have universal=1 set are producing poorly tagged wheels that should be warned about. Using Requires-Python instead of relying on universal=1
to bdist_wheel
, introducing a py
tag, all sounds like things that could be considered, but out of the scope of twine.
The Issue
With the sunset of Python 2, distribution maintainers should stop producing universal wheels with (ones with
py2.py3
tag).The solution is to drop
universal = 1
or turn that into 0 (https://github.com/pypa/packaging.python.org/issues/726) but it's easy to miss this bit. So this results in people adding things likepython_requires = >= 3.6
but still having thatpy2
tag in the wheels which is rather useless (and, well, incorrect).The same could happen to platform-specific wheels specifying ABI3 with tags like
abi3-py36
while havingpython_requires = >= 3.7
, for example.I propose improving the
twine check
command to detect such inconsistencies.