Ship wheels for PyPy

[ShaneHarvey]

I did not see PyPy discussed on https://github.com/pyca/cryptography/issues/5771 or https://github.com/pyca/cryptography/issues/5799.

Background: PyMongo currently relies on cryptography for a number of optional features. We test PyPy on Debian 9.2 and unsurprisingly these tests now fail when attempting to install cryptography because rust is not available on our test machine. PyPy always builds cryptography at install time because there are no PyPy specific wheels:

[2021/02/09 20:36:06.429]   running build_rust
--
[2021/02/09 20:36:06.429]   error: Can not find Rust compiler
[2021/02/09 20:36:06.429]       =============================DEBUG ASSISTANCE=============================
[2021/02/09 20:36:06.429]       If you are seeing a compilation error please try the following steps to
[2021/02/09 20:36:06.429]       successfully install cryptography:
[2021/02/09 20:36:06.429]       1) Upgrade to the latest pip and try again. This will fix errors for most
[2021/02/09 20:36:06.429]          users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
[2021/02/09 20:36:06.429]       2) Read https://cryptography.io/en/latest/installation.html for specific
[2021/02/09 20:36:06.429]          instructions for your platform.
[2021/02/09 20:36:06.429]       3) Check our frequently asked questions for more information:
[2021/02/09 20:36:06.429]          https://cryptography.io/en/latest/faq.html
[2021/02/09 20:36:06.429]       4) Ensure you have a recent Rust toolchain installed:
[2021/02/09 20:36:06.429]          https://cryptography.io/en/latest/installation.html#rust
[2021/02/09 20:36:06.429]       5) If you are experiencing issues with Rust for *this release only* you may
[2021/02/09 20:36:06.429]          set the environment variable `CRYPTOGRAPHY_DONT_BUILD_RUST=1`.
[2021/02/09 20:36:06.429]       =============================DEBUG ASSISTANCE=============================
[2021/02/09 20:36:06.429]   ----------------------------------------
[2021/02/09 20:36:06.430]   ERROR: Failed building wheel for cryptography

I know we have a number of temporary options (pin to 3.3, use CRYPTOGRAPHY_DONT_BUILD_RUST) and we can fix this for real by installing rust >=1.45. However, are there plans to improve the PyPy story here? Is it possible to ship wheels for PyPy?

It might also be helpful to include a PyPy specific error message since upgrading pip will not help in this case.

[reaperhulk]

Unfortunately PyPy has no stable ABI (except within a release series, with tags like pp73). There was some conversation about this in another open issue (https://github.com/pyca/cryptography/issues/5696#issuecomment-763132517).

It's not really feasible for us to maintain wheel build infrastructure for an ever-increasing set of PyPys and we're reluctant to only ship wheels on the latest PyPy since that results in significant confusion when we drop a wheel build or when PyPy does a new release and we don't immediately ship a wheel for it.

We're definitely not opposed to wheels here, but PyPy probably needs to commit to some abi3 style stable ABI that we could use to ship wheels to avoid the problems I've described.

[chrysn]

It would help if cryptography could keep a conservative MSRV.

Distributions seem to be relatively willing to update Rust versions (eg. Debian stretch's Rust was updated to 1.41). That, paired with a slow-bumping MSRV should make installing a suitable Rust compiler easier way before any ABI commitment would happen.

[edit: For my own CI needs at aiocoap I install Rust via rustup for the time being while the pypy containers don't ship a Rust 1.45]

[tiran]

It would help if cryptography could keep a conservative MSRV.

Distributions seem to be relatively willing to update Rust versions (eg. Debian stretch's Rust was updated to 1.41). That, paired with a slow-bumping MSRV should make installing a suitable Rust compiler easier way before any ABI commitment would happen.

I agree, it would be helpful. The pyo3 crate requires a Rust 1.45 for some features like .strip_prefix(). Could you please open a request on https://github.com/PyO3/pyo3/ ? The project might be able to accommodate 1.41.

[tiran]

I have created https://github.com/PyO3/pyo3/issues/1420

[ShaneHarvey]

Thanks for the responses! I've opened a feature request for pypy to add a stable abi: https://foss.heptapod.net/pypy/pypy/-/issues/3397

[mattip]

The current situation is that it is no longer possible to build cryptography from source on most end-user platforms. This is unfortunate. Until PyPy gains a stable API, could cryptography distribute a single wheel for pypy3.7? I don't think this would cause much churn: right now pypy3.7 is really the only version that is viable (many projects have dropped support for python3.6, and there is no pypy3.8). New wheels do not have to be created for every PyPy release, only per-python-version.

[tiran]

The current situation is that it is no longer possible to build cryptography from source on most end-user platforms

Which end-user platforms are missing? With latest release, PyCA cryptography builds with standard OS packages on most platforms except for Alpine 3.11, CentOS 7, Ubuntu 16.04, and some retro hardware.

[mattip]

Ahh, sorry, I must have misunderstood the issue: I thought the problem was more widespread.

[tiran]

Release 3.4.0 was not compatible with Rust < 1.45, which broke building on Debian stable and Alpine 3.12. With https://github.com/PyO3/pyo3/issues/1420 and 3.4.5, Rust 1.41 is supported. 3.4.5 came out a week after 3.4.0.

[reaperhulk]

Will PyPy go to 7.4 (and therefore a pp74 tag) when they support Python 3.7? I'm not committing to anything, but want to be sure I understand what PyPy's current ABI looks like. Even without abi3 if it's stable within a major release series that's more manageable.

[tiran]

Does PyPI even allow uploads of PyPy binary wheels? AFAIK PyPI blocks uploads for some wheel tags.

[mattip]

Will PyPy go to 7.4 (and therefore a pp74 tag) when they support Python 3.7?

PyPy supports python 3.7 with the pp73 tag. There are no plans to change the pp73 tag in future releases, and when PyPy supports python3.8 the plan is to remain with the same pp73 tag.

Does PyPI even allow uploads of PyPy binary wheels?

Yes, NumPy and others support it.

[reaperhulk]

Okay, if that's the situation then the burden of shipping a wheel here isn't incrementally huge. I'll look into this a bit.

[reaperhulk]

It looks like there's a bit of nuance here. numpy-1.20.1-pp37-pypy37_pp73-manylinux2010_x86_64.whl suggests that pypy3.7 7.3 and pypy3.6 7.3 will require separate wheels. Is that the case?

[mattip]

numpy-1.20.1-pp37-pypy37_pp73-manylinux2010_x86_64.whl is the wheel for pypy37 which is python3.7. numpy-1.19.5-pp36-pypy36_pp73-manylinux2010_x86_64.whl is the wheel for pypy36 which is python3.6.

Since they target different versions of python, they require different wheels, just like on CPython. However they both use the same underlying RPython/cpyext APIs, which is described by the pp73 binary API tag.

In the distant past (two years ago), the binary API was not stable and so many point releases of the same version of python with PyPy would update the binary API, from pp43 all the way up to the current pp73. Although back then no-one was putting PyPy wheels up on PyPI, so it didn't really matter. Now that it does matter, PyPy is more careful about updating the binary API, so once a project creates wheels for a pypy version we try very hard not to break backward compatibility.

If cryptography were to release both pypy3.6 and pypy3.7 wheels that would be great. But right now pypy3.7 is really the only version that is viable (many projects have dropped support for python3.6, and there is not yet a pypy3.8). I hope that once PyPy3.8 is released, PyPy will have a stable API so a single wheel will suffice for pypy3.8+.

[reaperhulk]

pypy{36,37}_pp73-manylinux20{10,14} wheels are now uploaded. They are part of our release process so they'll be automatically uploaded going forward.

[mattip]

Thanks.

I should have looked at the related issues before editoralizing about this, I apologize to @tiran for adding any stress on top of what must have been a very stressful couple of weeks.

[tiran]

Thanks for your kind words. :1st_place_medal:

[altendky]

@reaperhulk, is there an expectation of building and publishing PyPy wheels for macOS and Windows as well? Honestly, I don't even get to use PyPy since I use Qt wrappers, but I do try to help expand PyPy support by including it in test matrices of whatever projects I can. If needed, I might be able to find some time to make PR.

[mattip]

I would suggest to hold off on macOS and Windows at least until the next release. There is a bug in macOS, and PyPy only provides a 32-bit windows version until the next release. People who want PyPy on macOS can get it (and I assume cryptography) via conda or homebrew.