Offer wheels for borg on pypi

[modem7]

Continuing on from https://github.com/borgbackup/borg/issues/6721 with new information and ideas:

Whilst still working with @grantbevis on https://github.com/borgmatic-collective/docker-borgmatic, I have revisited cibuildwheel.

Using github actions, it should be relatively straightforward to build a wheel for x86_64 and arm64, then upload to pypi.

Initial setup would look something like this.

And delivery to pypi can be achieved like this.

I have done a brief test on an Ubuntu VM to generate a x86_64 and aarch64 musllinux_1_1 wheel:

Test steps: Fresh Ubuntu VM Install pipx + git

git clone https://github.com/borgbackup/borg.git
cd borg
nano pyproject.toml

Add the following to the top of the file (I'm sure not all of these packages are required, some are historical for some other packages, so please feel free to optimise):

[tool.cibuildwheel]
before-all = "apk add -Uu acl-dev alpine-sdk attr-dev attr-dev bzip2-dev fuse-dev g++ libcrypto1.1 libffi-dev libssl1.1 libxxhash linux-headers lz4-dev musl-dev ncurses-dev openssl-dev py3-msgpack py3-packaging py3-pip py3-setuptools py3-setuptools_scm py3-wheel py3-xxhash python3-dev readline-dev sqlite-dev tree xxhash-dev xz-dev zlib-dev zstd-dev && python3 -m pip install -U pkgconfig pip setuptools wheel"
build = "cp311-musllinux_*"

[tool.cibuildwheel.linux]
archs = ["x86_64", "aarch64"]

run pipx run cibuildwheel --platform linux

Output:

root@borg:/home/modem7/borg# ls wheelhouse/
borgbackup-2.0.0b5.dev276+g3f2aac8f.d20230222-cp311-cp311-musllinux_1_1_aarch64.whl  
borgbackup-2.0.0b5.dev276+g3f2aac8f.d20230222-cp311-cp311-musllinux_1_1_x86_64.whl

I'm sure much more modifications can be done to better suit the project, but it seems like a quick win to implement, and automatically build and deploy.

@ThomasWaldmann and @grantbevis - I know we've discussed this before to a degree, but cibuildwheel has come quite some distance now, especially with being able to build musllinux.

Do either of you have any thoughts on the above?

[ThomasWaldmann]

Some features of borg require glibc and (IIRC) don't work on musllinux. Not sure what consequences there are (if any), if the build is on musllinux.

[modem7]

Some features of borg require glibc and (IIRC) don't work on musllinux. Not sure what consequences there are (if any), if the build is only musllinux.

So far, they've been working very well with our manual wheel builds. But it's cumbersome to manually do it each time as the arm builds take quite a while.

[modem7]

I've got a semi-working example here:

https://github.com/modem7/wheels

The only reason it's semi-working is that I'm unable to upload to pypi due to the package name (obviously).

You can download the resultant wheel file here if you wish to test: https://github.com/modem7/wheels/actions/runs/4247195587

Result of testing on x86_64 version of python:3.11.2-alpine3.17:

[ThomasWaldmann]

Guess I rather won't jump on alpine / musl.

There's enough to do with borg itself, so I don't want to care about libc "special effects and limitations".

Edit: as I said, consequences of building on alpine/musl are unclear to me. But considering that "musllinux" ends up in the wheel name, guess there might be some?

[bbx0]

Probably manylinux is a more general purpose approach to the glibc problem instead of using musl. It is already used by the dependencies of borg for their wheels on pypi.

For the docker part I recently setup an automated pipeline for vanilla borg, which compiles multi-arch and is quite fast due to GHA caching and parallelization. A new image for a borg is produced in ~30min covering all platforms (incl. ARM).

https://github.com/bbx0/container-borgbackup

• Builds all supported borg versions (1.1, 1.2 and beta 2.0) multi-platform in parallel (amd64, armv7, aarch64, ..)
• Respects the dependencies as used for the upstream standalone binary releases ..if I dissected the Vagrantfiles correctly :)
  • Debian 11 (borg-linuxnew64), python versions (3.7, 3.9, 3.11) and the pinned pip requirements
• Uses the "official" python docker images with automatic daily rebuild when the base image is updated (or when a new borg version is released)

Maybe it's an option to use this as the base image for borgmatic? This would solve your build dependencies and simplify the build chain. I appreciate borgmatic as well and can support the effort to migrate docker-borgmatic. There is really not much benefit for using python on Alpine with Docker, when there is no committed upstream support.

I'm also happy to move my pipeline under the borgmatic umbrella, if that is of any interest. Unless, there is a way to have an offical BorgBackup image, which of course would be the preferred option for all of us. :)