RFC: Drop PyPy

[h-vetinari]

It gives me no pleasure to write this, but the signs have been mounting for some time.

Since the introduction of PyPy in conda-forge ~4.5 years ago, we've migrated for 3.6, 3.7, 3.8 & 3.9, introduced windows support halfway through, and made hundreds of packages available pre-compiled for PyPy, or even PyPy-compatible upstream in the first place.

The lion's share of the work there was done by @mattip, one of the PyPy devs, who was truly tireless in getting this effort off the ground. Huge kudos for this work! But unsurprisingly, no single person can keep this up forever, and since the rest of cf/core does not have a lot of PyPy expertise (much less spare time), it all came down predominantly to Matti. When signs of fatigue showed up, I proposed to drop the least common architectures for PyPy https://github.com/conda-forge/conda-forge.github.io/blob/14deb66144d112a52d38b9ff62f6c6673a105fb7/community/minutes/2023-10-04.md?plain=1#L55-L58

This was rejected at the time, but the situation hasn't changed since then. The linked issue is a year old, but not much has changed since then. In particular, we have never migrated for PyPy 3.10, which is becoming an issue because many NEP-29 packages are starting to drop Python 3.9 support, and more importantly, we de-scoped PyPy from the numpy 2.0 migration because we concluded

@mattip: Maybe it is time to drop PyPy from future migrations. Packages that have been built should continue to work, so current users would not be suddenly cut off, and maintainers would not have to deal with the extra headache. I don't see a PyPy3.10 migration happening, and 3.9 is being dropped in the scientific python ecosystem.

When this briefly came up in a core call a few months ago (not minuted AFAIR), dropping PyPy was already considered an option. I think it's time to make this official. If desired, we could draw this out until we close the numpy 2.0 migration, but in the meantime I think we should announce that we're sunsetting support and stop the PyPy migrator.

No matter the outcome, again a huge thanks to @conda-forge/pypy3-6 @conda-forge/pypy-meta, and @mattip & @isuruf in particular! 👏 🫡

[hmaarrfk]

Are there alternatives to outright dropping PyPy? I like encouraging experimental things, and without an easy way to install PyPy packages, it would make experental things like this impossible to test at wider scale.

[h-vetinari]

Existing artefacts will of course remain installable. After the numpy 2.0 migration closes I don't see a good way to keep building PyPy (well, still technically possible if we keep numpy in the python-zip, but that's painful). At the absolute longest, we'd keep PyPy for anther ~14 month until python 3.9 as a version is dropped in late 2025, but given how support is slowly bitrotting without much attention, I don't see much tangible benefit; to me it seems like just postponing the inevitable. 🤷

Perhaps other people think differently of course!

[hmaarrfk]

I is my understanding that as of today, the pypy package for numpy doesn't exist.

We can remove the zip, and work with mattip to help eventually build out numpy 2 support.

[h-vetinari]

and work with mattip to help eventually build out numpy 2 support.

You should ask him if he's up for that. I did speak with him personally a while ago, and my understanding was that he has no time/energy for shepherding things in conda-forge anymore.

[hmaarrfk]

That makes sense.

I guess my point is that even if you remove the numpy/python zip it won’t hurt PyPy given that no numpy package exists for it today.

[mattip]

I can slowly push things forward if there is a need. I do appreciate the enormous amount of effort everyone has put in to PyPy support on conda-forge. My impression is that the downloads for PyPy and PyPy packages does not justify the effort. I would be happy to discover I am mistaken.

[hmaarrfk]

I just keep seeing people praise pypy on hacker news. Actually your question (I presume it is you) https://news.ycombinator.com/item?id=36940871 was encouraging to me 1 year ago.

[h-vetinari]

To be clear, I've always been in favour of keeping PyPy (it's a huge pity IMO to throw out all the work that went into it), but that also means migrating for 3.10 and fixing the issues that arise there. And without support from the PyPy devs, we're going to be stuck. Historically that has only been Matti, and it isn't fair to put all that on his shoulders.

Also, with CPython gaining a JIT and the faster-python project, the main selling points of PyPy are being diluted. Arguably we'd have a bigger bang for the buck to put that migration effort into 3.13 builds with JIT and free threading support.

[hmaarrfk]

From what I saw, the problems typically arise when users try to use parts of the C APi that aren't exposed Hy PyPy.

It would be interesting for me to drop PyPy3.9 and start a PyPy3.10 migration. I personally don't see any big challenges until we hit the packages that have C extensions (similar to the ones that are already stuck).

From a testing perspective, keeping a distinct build that doesn't end with cpython would also help.

Ultimately, with Matti doesn't want the increased testing and support surface, I'm ok with no longer supporting it at conda-forge.

[h-vetinari]

This was discussed in the core call today, where we agreed to make an announcement about the intent to drop PyPy in roughly 6-8 weeks, so that people can come back to us if they depend on this in some crucial way. To fully reverse course though, someone would have to promise competent support (which means either developers from PyPy, or funding that effort somehow).

[mattip]

I am happy to go either way: if there is a real need being served by conda-forge offering PyPy I can continue to support it. My impression is that it was a nice experiment but did not become critical infrastructure.

Again: I would like to express appreciation to the conda-forge team and all those who took up this challenge, made basic infrastructure changes to allow another python implementation, patiently triggered resource-consuming migrations, and graciously helped debug incompatibilities in packages.

[h-vetinari]

@mattip: if there is a real need being served by conda-forge offering PyPy I can continue to support it. My impression is that it was a nice experiment but did not become critical infrastructure.

In light of this offer (thank you 🙏), I could reformulate the announcement in #2259 differently. Something like:

[DRAFT] There is a possibility that we receive the necessary support from the PyPy developers, if it turns out that enough people depend on PyPy support in conda-forge for it to be worth their time. As such, please speak up if you fall into this category.

You could obviously also ask for feedback on this through the various communication channels on PyPy side.

[mattip]

I think that draft is good the way it is. Once the PR is merged, I will change the link in this blog post to your published post.

https://pypy.org/posts/2024/08/conda-forge-proposes-dropping-support-for-pypy.html

[BrenBarn]

It's unfortunate to hear of this, but I certainly understand the reasons for it. I don't use PyPy for anything important, but occasionally experiment with it. I think the biggest bummer here is that, for people doing that kind of thing, having PyPy available in conda-forge was one of the easiest ways to make use of it, since it smoothly integrated into the environment creation process. Its removal will make it harder to access, and possibly lead to even less use of it over time. Hopefully at some point in the future it can make a glorious comeback! :-)