"undefined symbol: PyUnicode_AsUTF8AndSize" runtime linker error under PyPy 7.3.5

[amirouche]

blake3-py works with cpython 3.9, but it does not work with PyPy 7.3.5.

============================================================================================ test session starts =============================================================================================
platform linux -- Python 3.7.10[pypy-7.3.5-final], pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /home/amirouche/.cache/pypoetry/virtualenvs/asyncio-foundationdb-MsScwi0I-py3.7/bin/pypy3
cachedir: .pytest_cache
rootdir: /home/amirouche/src/python/found
plugins: cov-2.12.1, pylama-7.7.1, asyncio-0.15.1
collected 0 items / 1 error

=================================================================================================== ERRORS ===================================================================================================
_________________________________________________________________________________________ ERROR collecting tests.py __________________________________________________________________________________________
ImportError while importing test module '/home/amirouche/src/python/found/tests.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../.cache/pypoetry/virtualenvs/asyncio-foundationdb-MsScwi0I-py3.7/site-packages/_pytest/python.py:578: in _importtestmodule
    mod = import_path(self.fspath, mode=importmode)
../../../.cache/pypoetry/virtualenvs/asyncio-foundationdb-MsScwi0I-py3.7/site-packages/_pytest/pathlib.py:524: in import_path
    importlib.import_module(module_name)
/usr/lib/pypy3/lib-python/3/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1015: in _gcd_import
    ???
<frozen importlib._bootstrap>:992: in _find_and_load
    ???
<frozen importlib._bootstrap>:976: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:686: in _load_unlocked
    ???
../../../.cache/pypoetry/virtualenvs/asyncio-foundationdb-MsScwi0I-py3.7/site-packages/_pytest/assertion/rewrite.py:170: in exec_module
    exec(co, module.__dict__)
tests.py:11: in <module>
    from found import bstore
found/bstore.py:24: in <module>
    from blake3 import blake3
../../../.cache/pypoetry/virtualenvs/asyncio-foundationdb-MsScwi0I-py3.7/site-packages/blake3/__init__.py:1: in <module>
    from .blake3 import *
E   ImportError: /home/amirouche/.cache/pypoetry/virtualenvs/asyncio-foundationdb-MsScwi0I-py3.7/site-packages/blake3/blake3.pypy37-pp73-x86_64-linux-gnu.so: undefined symbol: PyUnicode_AsUTF8AndSize

[oconnor663]

Interesting. I'm not very experienced with the details here (PyO3 does a very good job of just Making Things Work), but I wonder if this is related to this comment in PyO3:

// PyUnicode_AsUTF8AndSize only available on limited API from Python 3.10 and up.

These declarations might also be relevant? @davidhewitt do you think I should report this upstream on the PyO3 repo?

[davidhewitt]

The FFI declaration you pick out is correct. Looks like in PyO3 0.12 there was a bug with this declaration where it lacked the PyPy name. If you upgrade pyo3 things should work.

https://github.com/PyO3/pyo3/blob/0fdd09afda0c60562ced9df8734778de8555eb10/src/ffi/unicodeobject.rs#L140

[amirouche]

Thanks!

[oconnor663]

Awesome. I'll release a version bump.

[oconnor663]

Just released version 0.2.0 and confirmed that pip install blake3 under PyPy gives a working library.

[oconnor663]

I'd like to add PyPy to the regular CI test runs, but it looks like maturin develop doesn't work under PyPy. (It runs, but then trying to import blake3 doesn't find anything.) I might play around with hacking something toget to get the local build installed, but please let me know if I've missed anything obvious.

[davidhewitt]

Hmm it's very possible there could be issues with maturin develop on PyPy? If so, worth reporting upstream.

In our PyO3 CI we install a test module using tox, which works ok. https://github.com/PyO3/pyo3/tree/main/examples/pyo3-pytests

[oconnor663]

Thanks, I've filed https://github.com/PyO3/maturin/issues/595.