Open lpsinger opened 1 month ago
Thanks for working on this @lpsinger. The current behavior certainly doesn't seem quite right - we should never be producing something with a tag that pip
won't understand.
Related to gh-624. If I understand correctly, with the build option discussed in gh-624 you can disable asking for limited API usage, which will then produce a tag like pp310-pypy310_pp73-linux_x86_64
that can be installed. This PR is a little different in that you want to avoid having to think about using that build option at all, still use the limited C API, but produce a tag like pp310-pypy310_pp73-linux_x86_64
. Is that right? If so, that seems reasonable to me, since it reduced friction and the end result should be the same as erroring out and forcing the user to specify -C-Dpython.allow_limited_api=false
.
CI is a little unhappy, the tests need updating to match:
FAILED tests/test_tags.py::test_tag_stable_abi - AssertionError: assert 'pp310-pypy310_pp73-linux_x86_64' == 'pp310-abi3-linux_x86_64'
- pp310-abi3-linux_x86_64
+ pp310-pypy310_pp73-linux_x86_64
FAILED tests/test_wheel.py::test_limited_api - AssertionError: assert 'pypy310_pp73' == 'abi3'
This PR is a little different in that you want to avoid having to think about using that build option at all, still use the limited C API, but produce a tag like
pp310-pypy310_pp73-linux_x86_64
. Is that right?
That's correct. I'm looking to avoid having to add this line to projects that use the limited API and cibuildwheel: https://github.com/nasa-gcn/hpx/blob/main/pyproject.toml#L81-L84
CI is a little unhappy, the tests need updating to match:
I pushed a change for that; let's see if it fixes the CI.
My understanding is that PyPy supports the limited API but does not use a dedicated filename suffix for extension modules compiled for the limited API. The Python support in Meson implies the same understanding https://github.com/mesonbuild/meson/blob/cfd57180eef9036c7167c5682b9f3055a540fccc/mesonbuild/scripts/python_info.py#L103-L107
What I am not sure about is whether, when using the limited API, PyPy implements a stable ABI, and thus the resulting wheels should be taggeg abi3
or not. If this is the case, the problem being solved here is a bug in pip
and not something we need to work-around in meson-python.
@mattip Can you confirm that my understanding is correct?
I added the tool.meson-python.limited-api
setting based on the assumption that PyPy supports the limited API without a dedicated module filename suffix and thus it is not sufficient to look at the filenames of the installed extension modules to determine whether a package uses the limited API only. If the assumption is wrong and PyPy does not support the limited API or supports it without exporting a stable ABI, the setting could in principle be removed.
PyPy does not implement a stable ABI. Wheels for PyPy should not be tagged abi3
. The output of pip
's get_supported
is correct.
Thanks for the clarification @mattip. My reasoning for implementing the tool.meson-python.limited-api
setting is wrong. I think we should remove this setting and rely on the filename suffix of the installed extension modules to decide the ABI tag for the created wheel. We would still accept the parameter for backward compatibility but emit a deprecation warning when we encounter it. @rgommers, what do you think?
One of the reasons to remove the tool.meson-python.limited-api
setting is that it is misnamed. When I introduced it, I thought that using the limited API always results in a stable ABI, and thus in wheels to be tagged with the abi3
tag. However, this is not the case on PyPy. Therefore, mixing the two concepts is not a good idea. To my defense, the CPython documentation does not make this distinction at all.
You can’t do that on windows. No suffix there.
In scikit-build-core, I use wheel.py-api, which is the wheel tag and you can get both limited api and pythonless wheels from that, see https://scikit-build-core.readthedocs.io/en/latest/configuration.html#customizing-the-output-wheel.
Setting something like cp311 is ignored on CPython < 3.11, PyPy, and free-threaded Python.
IIRC, on windows there is no suffix specific to abi3
extension modules but there is a Python ABI specific suffix and the general .pyd
suffix. I was thinking of mapping the .pyd
suffix to the abi3
wheel tag. This is not fool proof, but I don't expect it to be a problem in practice. I need to go back and check my reasoning from when I implemented the tool.meson-python.limited-api
setting.
@henryiii Thanks for the pointer to how scikit-build-core solves this. However, it is not clear to me how you handle the case when the user requires the stable ABI tag abi3
but the package is built for PyPy. Do you simply override the user choice in this case?
My impression is that the current build option is fine, since it works as designed for CPython. For PyPy, proceeding with the regular minor-version-specific tag as done in this PR would seem reasonable to me. The question I have is whether we should be emitting a warning when doing that.
However, this is not the case on PyPy. Therefore, mixing the two concepts is not a good idea.
PyPy could implement it in the future in principle (it's just a missing feature now), so I wouldn't attach too much value to this since conceptually it does make sense.
Moreover, PyPy is probably going to fade away pretty quickly over the next 1-2 years - there is unlikely to be a 3.11 release for PyPy, so next year when Python 3.10 starts being dropped that's the end of it. Conda-forge is also starting to drop PyPy builds because of maintenance issues and no real future. Hence deprecating because of PyPy isn't needed imho.
The way selecting cp311
works is that it's only applicable on CPython with GIL 3.11+. Any older version or other interpreter (including PyPy) ignores this choice. (It wasn't quite right for PyPy until the most recent release, when I was working on free-threading I cleaned it up a little). A user is expected to skip requesting it CMake like this:
if(NOT "${SKBUILD_SABI_COMPONENT}" STREQUAL "")
python_add_library(some_ext MODULE WITH_SOABI USE_SABI 3.11 ...)
else()
python_add_library(some_ext MODULE WITH_SOABI ...)
endif()
(USE_SABI also sets the Limited API define) Though I haven't checked to see exactly how CMake's FindPython module handles it, it might do the right thing (IMO which is to ignore this setting if it doesn't apply).
Not setting the define just means you are not limited to the Limited API, so shouldn't hurt user code.
Pip does not support installing PyPy wheels with the abi3 tag. If you build a wheel for PyPy and set the ABI tag to
abi3
, pip will not be able to install it.Full output: