Apple Silicon binary not included in current PyPI package

[AaronC81]

Running pip install pygit2 on my M1 Pro MacBook completes without any errors. But if I then try to import it, I get the following exception:

aaron@aaron-wyvern ~ % python
Python 3.10.0 (default, Nov  1 2021, 21:07:02) [Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pygit2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/aaron/.asdf/installs/python/3.10.0/lib/python3.10/site-packages/pygit2/__init__.py", line 30, in <module>
    from ._pygit2 import *
ImportError: dlopen(/Users/aaron/.asdf/installs/python/3.10.0/lib/python3.10/site-packages/pygit2/_pygit2.cpython-310-darwin.so, 0x0002): tried: '/Users/aaron/.asdf/installs/python/3.10.0/lib/python3.10/site-packages/pygit2/_pygit2.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/_pygit2.cpython-310-darwin.so' (no such file), '/usr/lib/_pygit2.cpython-310-darwin.so' (no such file)

...with the most important bit seeming to be incompatible architecture (have 'x86_64', need 'arm64e').

If I instead use pip install pygit2 --no-binary pygit2, the package works as expected, as long as I have libgit2 installed on my system.

Would it please be possible to include a pre-built macOS ARM binary in the PyPI package?

[skshetry]

pygit2 does build a universal2 wheel that should work with both architectures. Reading through https://github.com/pandas-dev/pandas/issues/39053, it seems that we need to build wheels on macOS 11 or later. We use macos-latest which is macos-10.15.

https://github.com/libgit2/pygit2/blob/76f84c07671417ddda3fbebc538c19bc25c3a9a9/.github/workflows/wheels.yml#L37

I don't have a M1 machine to test, but you can try it in your fork and check the built wheel from Github Actions if it works.

[AaronC81]

Thanks for the pointer!

I made a fork and swapped out macos-latest for macos-11, but the generated universal2 wheel still doesn't seem to work on my system, with a different error about a missing symbol:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/aaron/.asdf/installs/python/3.10.0/lib/python3.10/site-packages/pygit2/__init__.py", line 30, in <module>
    from ._pygit2 import *
ImportError: dlopen(/Users/aaron/.asdf/installs/python/3.10.0/lib/python3.10/site-packages/pygit2/_pygit2.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '_git_annotated_commit_free'

(I am using MacOS 12.0 21A344, if that could make any difference?)

Happy to keep looking into this once I have some free time, but not sure when that might be.

[efiop]

Can confirm this problem on m1 with macos 11. Seemed like a problem with https://github.com/pypa/pip/issues/9506#issuecomment-766423055 , but upgrading doesn't help. Trying to figure out what is going on.

[simoncozens]

OK, I've tried to do some digging into this, and it's going to be tricky to build an M1 wheel via CI. This is because:

• We have no M1 runners on Github, so we need to build a universal wheel on macOS-11.
• To build a universal wheel, we need to build a universal libgit2 and link against that.
• To build a universal libgit2, we need to build a universal libssh2 and link against that.
• libssh2 links against openssl. OpenSSL is installed via homebrew. Homebrew doesn't support universal building. So we have to build openssl from source as a universal binary, and link against that.

I'm working on the building part, but even if I succeed, relocating this chain of shared library dependencies into the wheel is probably going to be beyond my knowledge. I feel like statically linking at least some of them might be more manageable...

[jdavid]

Try:

LIBSSH2_VERSION=1.10.0 LIBGIT2_VERSION=1.3.0 sh build.sh wheel

This will build libssh2, libgit2, and then a pygit2 wheel.

So you need to add OPENSSH_VERSION=x.y.z to build.sh

[simoncozens]

It's a bit more complicated than that, because universal builds are not the default: you need to pass CMAKE_OSX_ARCHITECTURES to the cmake builds, you do need to build openssl twice and lipo them together, and so on. Anyway, it looks like I got it working. PR incoming, hopefully.