Add wheel support for musllinux (alpine) in accordance with PEP-656

[lfvjimisola]

On alpine pip install pygit2 does not work due to wheel not supporting musl (only glibc?).

Since alpine is very common in container enviroments (Kubernetes) it would be great if the project would fulfull PEP 656 – Platform Tag for Linux Distributions Using Musl.

Current workarounds are quite tedious, e.g.

1. manually compiling the needed wheel
2. use distro package (apk) py3-libgit2 and sync version with pygit2 (alpine is rarely on latest version)

[lfvjimisola]

@jdavid Sorry for tagging you but this ticket has not gotten any attention since I wrote it over 6 months ago. Just want to make sure that it has not gone unnoticed.

[jdavid]

In the musl branch I've changed the CI action to build wheels for musl as well, but it fails, https://github.com/libgit2/pygit2/actions/runs/7336708081/job/19976443113 This is something for a user of musl to contribute (I don't use musl).

[jimisola]

I'm currently afk. Tried to see using my phone but didn't get far. Just to confirm, is musl wheel build using alpine as imagine?

(I saw something about Ubuntu 22.04)

[lfvjimisola]

@jdavid The issue is that alpine comes with busybox's version of coreutils such as wget (not standard wget etc).

wget -N is used in build.sh and -Nis not supported by busybox's wget

Sadly, CIBW_BEFORE_BUILD: apk add wget won't work since it affects all builds and I can't find a way to find to it using cibuildwheel.

curl is normally a better option with alpine (musl), but if that's not an option then when building for alpine/musl install wget as you do for apt-get here:

https://github.com/libgit2/pygit2/blob/bf2d7542846fc444b86d02d44536103b27b9141f/build.sh#L62

I'm not sure that this change alone will mke it work since the build script contains a lot of downloaded source, manual builds etc (e.g. here).

Will the CI work if I fork the musl branch? Or does the changes have to be made in the libgit2/pygti2 project?

[jdavid]

Yes it's okay to switch to curl. I think so, the CI should run on the musl branch in your fork.

[jimisola]

@jdavid I kept wget but had do some other changes for the build to work (I've disabled all tests for now).

Successfully creating a wheel is created for musllinux: https://github.com/jimisola/pygit2/actions/runs/7479000908/job/20355210682#step:6:10973

I could use input on the following matter:

CIBW_REPAIR_WHEEL_COMMAND_LINUX is set as

CIBW_REPAIR_WHEEL_COMMAND_LINUX: "LD_LIBRARY_PATH=/project/ci/lib64 auditwheel repair -w {dest_dir} {wheel}"

but for musllinux it needs to be lib instead of lib64, i.e.:

CIBW_REPAIR_WHEEL_COMMAND_LINUX: "LD_LIBRARY_PATH=/project/ci/lib auditwheel repair -w {dest_dir} {wheel}"

There does not seem to be a way to do this easily using environment variables as the the LD_LIBRARY_PATH has to be depending on whether build is for musllinux or manylinux. I haven't found a way to determine the current build wheel "image" as this would allow us to set it during the build phase of each wheel, either directly setting LD_LIBRARY_PATH (and not include it in CIBW_REPAIR_WHEEL_COMMAND_LINUX) or e.g. by setting LD_LIBRARY_PATH_CURRENT for each build wheel and then use:

CIBW_REPAIR_WHEEL_COMMAND_LINUX: "LD_LIBRARY_PATH=$LD_LIBRARY_PATH_CURRENT auditwheel repair -w {dest_dir} {wheel}"

However there is a way to it using configuration overrides but those can only be specified in pyproject.toml.

See: https://cibuildwheel.readthedocs.io/en/stable/options/#overrides

The LD_LIBRARY_PATH issue would then be resolved as:

[tool.cibuildwheel.linux]
repair-wheel-command = "LD_LIBRARY_PATH=/project/ci/lib64 auditwheel repair -w {dest_dir} {wheel}"

[[tool.cibuildwheel.overrides]]
select = "*-musllinux*"
repair-wheel-command = "LD_LIBRARY_PATH=/project/ci/lib auditwheel repair -w {dest_dir} {wheel}"

Would you be ok to move the all configuration to pyproject.toml as per: https://cibuildwheel.readthedocs.io/en/stable/options/#configuration-file

It would allow for mine fine-grained configuration, if needed, in other areas as well, e.g. different build.sh depending on platform.

[jimisola]

This configuration is pyproject.toml would replace the configuration in wheel. It can be simplified further, e.g. only LIBGIT2 environment variable is different for macos compared to the default.

[build-system]
requires = ["setuptools", "wheel"]

[tool.cibuildwheel]
skip = "pp3*"
test-skip = ""

archs = ["auto"]
build-frontend = "default"
config-settings = {}
dependency-versions = "pinned"
environment = {LIBGIT2_VERSION="1.7.1", LIBSSH2_VERSION="1.11.0", OPENSSL_VERSION="3.1.3", LIBGIT2="/project/ci"}
environment-pass = []
build-verbosity = 0

before-all = "sh build.sh"
before-build = ""
repair-wheel-command = ""

manylinux-x86_64-image = "manylinux2014" # set per default
manylinux-aarch64-image = "manylinux2014" # set per default

[tool.cibuildwheel.linux]
repair-wheel-command = "LD_LIBRARY_PATH=/project/ci/lib64 auditwheel repair -w {dest_dir} {wheel}"
archs = ["x86_64", "aarch64"]

[[tool.cibuildwheel.overrides]]
select = "*-musllinux*"
repair-wheel-command = "LD_LIBRARY_PATH=/project/ci/lib auditwheel repair -w {dest_dir} {wheel}"

[tool.cibuildwheel.macos]
environment = {LIBGIT2_VERSION="1.7.1", LIBSSH2_VERSION="1.11.0", OPENSSL_VERSION="3.1.3", LIBGIT2="/Users/runner/work/pygit2/pygit2/ci"}
repair-wheel-command = "DYLD_LIBRARY_PATH=/Users/runner/work/pygit2/pygit2/ci/lib delocate-wheel --require-archs {delocate_archs} -w {dest_dir} {wheel}"
archs = ["universal2"]