[rfc] native aarch64 builders

[dkwo]

Cross-compilation is subtle.

1. any mainteiners interested? @leahneukirchen @classabbyamp @paper42
2. any users interested in supporting this?
3. hardware suggestions

I'd be happy to donate some money to purchase hardware, if you think it's doable and enough people are interested.

[Eloitor]

I'm interested in this as a user. I would like to use void in my pinephone. I tried to create a template for osmin (an OSM app) but it failed to build for aarch64.

[classabbyamp]

you can already compile natively for any arch yourself

[dkwo]

Compiling natively is fine, but you'd have to rebuild lots of subtly miscompiled stuff first, like qmake, libtool, etc, and build stuff like sagemath yourself, while having aarch64 as a first-class citizen (i.e. native) would avoid all of this.

[Eloitor]

you can already compile natively for any arch yourself

Compiling in the pinephone is a nightmare

[leahneukirchen]

I think the main problem is getting adequate hardware as an aarch64 builder, as server-grade aarch64 systems are still quite expensive; and most of them cannot run armv7 code, so it can't be a builder for both archs.

Also this merely shifts the burden of fixing cross-compiling to the other architectures.

[tornaria]

I have a rpi 4 with 8 GB ram and I will start testing some math software on it. It would be nice to have good support, we have some key packages that won't cross compile (ntl). But if some cross-compiled packages in the build chain are broken, that complicates matters a lot.

It seems aarch64 is more important than i686 at this point (e.g. a few days ago I learned that sagemath is completely broken on i686, but nobody noticed -- still we spend a lot of energy fixing 32 bit issues and 8087 magic in particular since we can't even assume sse2)

If it's not possible to get aarch64 builders, is there a chance that some (few, key, carefully selected) packages can be natively compiled on a x86_64 builder using qemu instead of cross?

[leahneukirchen]

An rpi 4 is not fast enough to keep up as a full builder.

[dkwo]

But if some cross-compiled packages in the build chain are broken, that complicates matters a lot.

Exactly, see libtool, possibly qmake.

[dkwo]

@leahneukirchen How about an M1 Mac mini from Apple?

[leahneukirchen]

Doesn't run armv7 code either. Probably fast enough, may need external storage.

Not sure if the different page size is problematic.

[dkwo]

Page size affects a few packages (I assume that https://github.com/AsahiLinux/docs/wiki/Broken-Software also applies in the reverse way), among them zig; many have already been fixed. @leahneukirchen Would you be interested in receiving something like that as a gift, provided enough people contribute?

[paper42]

It is simple to modify xbps-src to use qemu, just an example:

--- a/common/xbps-src/shutils/chroot.sh
+++ b/common/xbps-src/shutils/chroot.sh
@@ -160,7 +160,7 @@ chroot_handler() {

     if [ "$action" = "chroot" ]; then
         $XBPS_COMMONDIR/chroot-style/${XBPS_CHROOT_CMD:=uunshare}.sh \
-            $XBPS_MASTERDIR $XBPS_DISTDIR "$XBPS_HOSTDIR" "$XBPS_CHROOT_CMD_ARGS" /bin/xbps-shell
+            $XBPS_MASTERDIR $XBPS_DISTDIR "$XBPS_HOSTDIR" "$XBPS_CHROOT_CMD_ARGS" /usr/bin/qemu-aarch64-static /bin/bash /bin/xbps-shell
         rv=$?
     else
         env -i -- PATH="/usr/bin:$PATH" SHELL=/bin/sh \
@@ -178,7 +178,7 @@ chroot_handler() {
             ${XBPS_ALT_REPOSITORY:+XBPS_ALT_REPOSITORY=$XBPS_ALT_REPOSITORY} \
             $XBPS_COMMONDIR/chroot-style/${XBPS_CHROOT_CMD:=uunshare}.sh \
             $XBPS_MASTERDIR $XBPS_DISTDIR "$XBPS_HOSTDIR" "$XBPS_CHROOT_CMD_ARGS" \
-            /void-packages/xbps-src $XBPS_OPTIONS $action $pkg
+            /usr/bin/qemu-aarch64-static /bin/bash /void-packages/xbps-src $XBPS_OPTIONS $action $pkg
         rv=$?
     fi

echo XBPS_CHROOT_CMD=bwrap >> etc/conf
./xbps-src binary-bootstrap -m maarch aarch64-musl
cp /usr/bin/qemu-aarch64-static maarch/usr/bin/
./xbps-src -m maarch pkg libtool
./xbps-src -m maarch pkg ntl

This assumes bubblewrap, qemu-user-static and binfmt are available on the host. It would have to be improved if it was meant to be merged to void-packages and it's very slow.

[leahneukirchen]

Would you be interested in receiving something like that as a gift, provided enough people contribute?

I don't have resources to run this myself locally atm.

[vincele]

RK3588 support is coming real soon now (tm). This will be faster than rpi4, and less expensive than apple HW.

[BikyAlex]

any users interested in supporting this?

I'm willing to buy a Honeycomb LX2K and run it 24/7, I have been looking into it for a few years. Would that fit the bill for compiling aarch64, aarch64-musl, armv7 and armv7-musl? It's a NXP LX2160A 16x A72 cores aarch64 CPU.

I would love to contribute to Void some processing horsepower for native building arm packages. Some packages, like libreoffice are marked as non-cross-compileable. But I don't know how I'd be sharing it with Void devs, I'd like to share Void LXC containers for each architecture, but if really necessary, I could share access to the box itself, but I'd have to split my network and put it in a DMZ or something.

[dkwo]

Can anyone clarify: does current Void's infrastructure (buildbot etc) allow adding native aarch64 builders, sponsored by @BikyAlex or others? is there any obstruction?

[Duncaen]

We can't just integrate some random never seen before users hardware into our build infrastructure.

[Duncaen]

This is not really an RFC and I fail to see the point of this, yes native build servers would be better, everyone agrees. We don't ask for donations and can't take donations, keeping issues like this open that looks like a public request for donations is misleading.

[dkwo]

How about shipping hardware to a maintainer? Can this be discussed, or is it also excluded similar to money donations?

It seems RK3588 could be good enough, and e.g. an Orange Pi 5 Plus board is sold on Amazon for 150$. I'd be willing to buy it and ship it to a maintainer, if that can be used as a builder.

[dkwo]

let me mention it also here: there's also the option of applying to oregon state university for the aarch64 builders. https://osuosl.org/services/aarch64/current-projects/#foss-projects chimera, gentoo and others do that.

[BikyAlex]

Huh, q66 applied for Oregon aarch64 builders? Nice. I really wish Void had a bit more love for aarch64. I would also be willing to buy SBCs and send them to contributors, but that will count as donations, and I doubt any of the people working on void want to get in legal trouble. Most (all?) of them are doing a work of love here.

That said, can't force anyone to just maintain aarch64 packages and the many that cannot cross-compile, will be found to be broken when compiling on aarch64.

[dkwo]

(of course, oregon's would be native builders.)