pypa / pip

The Python package installer
https://pip.pypa.io/
MIT License
9.53k stars 3.03k forks source link

Drop support for Python 3.8 #12989

Open notatallshaw opened 1 month ago

notatallshaw commented 1 month ago

Python 3.8 support, from the Python Foundation, is ending in a few weeks: https://endoflife.date/python

This is to track ending Python 3.8 support, see 3.7 end: https://github.com/pypa/pip/issues/11934 and https://github.com/pypa/pip/pull/11944

There is motivation as pip will not be able to vendor urllib3 2 until Python 3.9 support is dropped: https://github.com/pypa/pip/issues/12857. Which contains bugfixes/features that will not be backported.

Code of Conduct

ichard26 commented 1 month ago

Paging @hugovk for the pip downloads stats. My gut reaction is that dropping support for Python 3.8 right after EOL may be a bit too painful.

hugovk commented 1 month ago

From https://hugovk.github.io/pypi-tools/charts.html#pip:

image

PyPI downloads for the last 30 days, for all versions of pip shows 9.52% with Python 3.8:

❯ pypinfo --all --percent --markdown pip pyversion
Served from cache: False
Data processed: 5.73 GiB
Data billed: 5.73 GiB
Estimated cost: $0.03
python_version percent download_count
3.11 24.16% 63,984,763
3.10 18.94% 50,175,163
3.7 17.09% 45,261,471
3.9 13.80% 36,541,609
3.8 9.52% 25,211,245
3.12 6.35% 16,818,377
3.6 3.83% 10,134,044
2.7 3.48% 9,216,031
None 2.64% 7,004,762
3.13 0.20% 524,934
Total 264,872,399

Limiting to only downloads of pip==24, with the reasoning that users of pip==24 are most likely to upgrade to 24.3 and excluding those who never upgrade anyway, shows 1.35% for Python 3.8:

❯ pypinfo --all --percent --markdown pip==24 pyversion
Served from cache: False
Data processed: 7.45 GiB
Data billed: 7.45 GiB
Estimated cost: $0.04
python_version percent download_count
3.7 72.42% 29,422,847
3.11 8.86% 3,599,931
None 6.70% 2,721,770
3.9 4.90% 1,991,069
3.10 3.04% 1,237,178
3.12 2.61% 1,059,822
3.8 1.35% 547,122
2.7 0.11% 44,067
3.13 0.01% 4,304
3.6 0.01% 2,723
Total 40,630,833
A 3.7 aside Python 3.7 has already been dropped, but [Amazon Linux is still responsible for all the 3.7 downloads](https://dev.to/hugovk/why-are-there-still-so-many-downloads-for-eol-python-37-30cp): ```console ❯ pypinfo --limit 20 --percent --markdown pip==24 system distro pyversion Served from cache: False Data processed: 13.65 GiB Data billed: 13.65 GiB Estimated cost: $0.07 ``` | system_name | distro_name | python_version | percent | download_count | | ----------- | ---------------- | -------------- | ------: | -------------: | | Linux | Amazon Linux | 3.7 | 69.89% | 24,760,195 | | Linux | Ubuntu | 3.7 | 7.20% | 2,549,869 | | Linux | Ubuntu | 3.9 | 4.97% | 1,759,663 | | Linux | Ubuntu | 3.11 | 4.53% | 1,606,464 | | Linux | Debian GNU/Linux | 3.7 | 3.96% | 1,402,552 | | Linux | Ubuntu | 3.10 | 2.82% | 1,000,292 | | Linux | Ubuntu | 3.12 | 1.26% | 445,796 | | Windows | None | 3.7 | 0.95% | 338,158 | | Linux | Debian GNU/Linux | 3.12 | 0.95% | 335,066 | | Linux | Ubuntu | 3.8 | 0.87% | 306,601 | | Windows | None | 3.12 | 0.52% | 183,841 | | Linux | Debian GNU/Linux | 3.8 | 0.40% | 143,088 | | Linux | Alpine Linux | 3.7 | 0.31% | 108,246 | | Linux | CentOS Linux | 3.7 | 0.28% | 99,851 | | Linux | Debian GNU/Linux | 3.9 | 0.26% | 92,373 | | Linux | Debian GNU/Linux | 3.11 | 0.22% | 76,679 | | Windows | None | 3.10 | 0.17% | 59,970 | | Linux | Debian GNU/Linux | 3.10 | 0.17% | 58,698 | | Linux | Amazon Linux AMI | 3.7 | 0.14% | 50,665 | | Darwin | macOS | 3.7 | 0.14% | 48,986 | | Total | | | | 35,427,053 |
notatallshaw commented 1 month ago

Limiting to only downloads of pip==24

Does that mean pip versions 24.* or pip version 24.0?

hugovk commented 1 month ago

Hmm, I thought it meant all 24.* but it's more like 24.0. Here's numbers for each minor version:

24.0: 40.7m (last to support 3.7) | python_version | percent | download_count | | -------------- | ------: | -------------: | | 3.7 | 72.42% | 29,446,324 | | 3.11 | 8.97% | 3,645,590 | | None | 6.67% | 2,713,850 | | 3.9 | 4.84% | 1,969,068 | | 3.10 | 3.03% | 1,230,237 | | 3.12 | 2.60% | 1,058,185 | | 3.8 | 1.34% | 545,217 | | 2.7 | 0.11% | 44,393 | | 3.13 | 0.01% | 5,211 | | 3.6 | 0.01% | 2,725 | | Total | | 40,660,800 |
24.1: 0.2m | python_version | percent | download_count | | -------------- | ------: | -------------: | | 3.12 | 22.36% | 49,503 | | 3.11 | 16.97% | 37,565 | | 3.8 | 16.16% | 35,779 | | 3.10 | 15.41% | 34,106 | | 2.7 | 14.04% | 31,069 | | 3.9 | 7.25% | 16,039 | | None | 7.00% | 15,500 | | 3.13 | 0.66% | 1,452 | | 3.14 | 0.11% | 240 | | 3.5 | 0.05% | 105 | | Total | | 221,358 |
24.2: 167.2m | python_version | percent | download_count | | -------------- | ------: | -------------: | | 3.11 | 31.56% | 52,780,082 | | 3.10 | 27.36% | 45,751,785 | | 3.9 | 18.69% | 31,250,535 | | 3.8 | 12.22% | 20,433,838 | | 3.12 | 8.57% | 14,338,283 | | None | 0.68% | 1,131,623 | | 2.7 | 0.48% | 809,263 | | 3.13 | 0.32% | 537,260 | | 3.7 | 0.10% | 174,948 | | 3.6 | 0.02% | 31,955 | | Total | | 167,239,572 |
ichard26 commented 1 month ago

I realize that in the case of Python 3.7, we were in favour of dropping support for it promptly, but I really don't see any worthwhile benefits from dropping 3.8 soon. It'd be nice to fix some bugs in the 24 series for 3.8 users, and then we can revisit the question closer to 25.0.

For context, I grepped all of pip's source code for references to Python 3.8 for any code that we could clean up and I only got one hit in the resolvelib glue: https://github.com/pypa/pip/blob/348c428b065aa1a685cda69aabc663b8e0a9b5e7/src/pip/_internal/resolution/resolvelib/found_candidates.py#L28-L39

Of course, starting in Python 3.9, many built-in types are now subscriptable for typing purposes. Other than that, I'm not aware of anything major we'd be gaining. Python 3.9 is indeed the real kicker as that unblocks the upgrade to urllib3's 2.x series. (And then Python 3.10 as that's the last version to use the old metadata backend.)

notatallshaw commented 1 month ago

24.1: 0.2m

Can you please check for 24.1.2, both 24.0 and 24.2 didn't have patch releases, but 24.1 had two followup patch releases, and I'm guessing by the fact you're seeing 24.1 as 100x less downloads it's because the numbers are only for 24.1 and not 24.1.1 or 24.1.2: https://pypi.org/project/pip/#history

notatallshaw commented 1 month ago

I realize that in the case of Python 3.7, we were in favour of dropping support for it promptly, but I really don't see any worthwhile benefits from dropping 3.8 soon. It'd be nice to fix some bugs in the 24 series for 3.8 users, and then we can revisit the question closer to 25.0.

From my point of view I think the only compelling reason to drop older versions of Python is to support any vendored packages that don't 100% support older versions of Python. And in this case I think the most important example is urllib3 2.0, which pip can not vendor until it drops Python 3.9. And the longer pip waits for Python 3.8 to be dropped the more of a shock it will be to users if Python 3.9 is dropped quickly.

I don't think there is significant technical debt in pip from supporting older 3.x versions, and if it wasn't for pip needing to vendor libraries I would consider it appropriate to support old versions of Python until the CI broke.

ichard26 commented 1 month ago

I think a reasonable cutoff is pip 24 is the last series to support Python 3.8. Not particularly meaningful, at least hey we can point "it's a major release, y'all happy semvar folks?!"

I'm only half-joking :)

hugovk commented 1 month ago

24.1: 0.2m

Can you please check for 24.1.2, both 24.0 and 24.2 didn't have patch releases, but 24.1 had two followup patch releases, and I'm guessing by the fact you're seeing 24.1 as 100x less downloads it's because the numbers are only for 24.1 and not 24.1.1 or 24.1.2: pypi.org/project/pip#history

Yep, that sounds about right, 24.1.2 is bigger but still fairly low:

24.1.1: 0.2m | python_version | percent | download_count | | -------------- | ------: | -------------: | | 3.10 | 21.79% | 48,633 | | 3.12 | 19.79% | 44,168 | | 3.8 | 17.69% | 39,478 | | 3.11 | 14.72% | 32,862 | | 2.7 | 13.94% | 31,119 | | 3.9 | 5.85% | 13,057 | | None | 5.77% | 12,871 | | 3.13 | 0.41% | 910 | | 3.5 | 0.05% | 108 | | 3.7 | 0.00% | 3 | | Total | | 223,209 |
24.1.2: 0.5m | python_version | percent | download_count | | -------------- | ------: | -------------: | | 3.12 | 18.93% | 86,180 | | 3.10 | 18.45% | 83,981 | | None | 16.27% | 74,055 | | 3.9 | 14.67% | 66,780 | | 3.8 | 13.56% | 61,759 | | 3.11 | 10.85% | 49,383 | | 2.7 | 7.00% | 31,885 | | 3.13 | 0.24% | 1,106 | | 3.5 | 0.02% | 111 | | 3.14 | 0.01% | 56 | | Total | | 455,296 |
notatallshaw commented 1 month ago

Yep, that sounds about right, 24.1.2 is bigger but still fairly low:

That's fascinating how starkly different orders of magnitude the 24.1 releases are, it really makes me wonder what dominates download numbers and how useful they actually are from a project maintenance perspective.

I think a reasonable cutoff is pip 24 is the last series to support Python 3.8. Not particularly meaningful, at least hey we can point "it's a major release, y'all happy semvar folks?!"

I'm only half-joking :)

I guess we take a look again closer to the 25.0 release? And if there isn't a truststore release and vendor before 24.3, as I mention in https://github.com/pypa/pip/issues/12941#issuecomment-2408616377, I think it would be quite helpful to some users to do one more Python 3.8 release with that new version of truststore.

hugovk commented 1 month ago

For context, I grepped all of pip's source code for references to Python 3.8 for any code that we could clean up and I only got one hit in the resolvelib glue:

Yeah, I didn't see much either, but there are a lot of typing changes from Ruff/pyupgrade, something like 200 files and 1,200 lines, for example: https://github.com/pypa/pip/compare/main...hugovk:pip:rm-3.8?expand=0

potiuk commented 1 month ago

Small comment from the user point of view.

There is one more benefit from dropping 3.8. It's good way to help the ecosystem to move to Python 3.9 quicker. Even if pip itself is not affected, generally moving faster to non-EOL version of Python is good for multiple reasons - including security reasons. Imagine someone finds a critical security issue in Python in March 2025 and it gets fixed only in 3.9+. People who would stil be using Python 3.8 are vulnerable, and with security, it's important to limit the damage - so the less number of people still use Python 3.8, the better.

As the default frontend used by many - pip has a tremendous impact on the whole ecosystem, and it has a built-in "please upgrade" warning displayed prominently that in many times actually make people upgrade. Using that power to drive the ecosystem to be safer, is pretty noble and good reason to drop 3.8 in 25.* version of pip - or even earlier if that would be accepted.

pfmoore commented 1 month ago

I don't think pip should be pushing users to upgrade their Python installation - that feels backwards to me. I'm happy with our current policy of dropping support for Python versions when usage (as measured by pip download figures for that version) gets low enough that we expect the impact to be minimal.

If the core Python team wants to push users to adopt newer versions faster, that's something they should do themselves.

potiuk commented 1 month ago

If the core Python team wants to push users to adopt newer versions faster, that's something they should do themselves.

Sure - if that's the maintainer positions, I just thought it's worth to share that perspective where the whole ecosystem works together - I am now working with @sethmlarson on Airflow Supply Chain security improvements and reviews with the goal of making it then possible to apply to the whole PyPI ecosystem gets benefit of what we do, and recently gave a talk at Airflow Summit Keynote about it called "Security United" where together with Alpha-Omega we stressed how important it is where everyone works together on improving security, regardless if it is "our" or "their" problem, so I thought I might mention that having pip becoming part of the effort might also be a good idea.

But if maintainers of pip think it's "not their problem", it might well be a good decision from their side, just wanted to mention you might look at it from wider perspective.

pfmoore commented 1 month ago

It’s not so much a matter of not our problem, more a case that I’d want the core team to take the lead on any effort to get people to upgrade Python more aggressively.

potiuk commented 1 month ago

It’s not so much a matter of not our problem, more a case that I’d want the core team to take the lead on any effort to get people to upgrade Python more aggressively.

Sure. I acknowldged your right to make that decision, provided different perspective and have shown and example where others think differently and act for the good of the whole ecosystem in principle. Just so you see this can be at least taken it into consideration when you make decisions about your project. No more, no less.

But of course it's you who decide where you spend your energy and efforts, and you have the right to do so - it's your call of course not mine. And it's not that I agree or disagree with it, it's not for me to decide, still I think it's worth knowing that others are thinking differently.

pradyunsg commented 1 month ago

FWIW, I should also note that what the numbers look like for pip install pip --upgrade etc isn't a particularly informative data point to be using here -- we want something akin to "downloads from PyPI, grouped by installer.version and platform, where installer.name = pip", and not "counts of downloads of pip's own wheels and sdists".

Last I checked, that was on the order of terrabytes for the query but I don't remember the timescale I was querying (point is:I couldn't do it on the free tier). It does make sense that these costs match with pypi.org's exponential growth tho.

ichard26 commented 1 week ago

I have a local patch that drops Python 3.8 support with pyupgrade and manual clean-up done. I haven't PRed it as it needs a bit more work, and it's going to be very merge-conflict prone so I'll wait until we decide on a timeline.