Arm64 - take2

[pat-s]

• Only running "build images" WF for testing purposes

[pat-s]

@techknowlogick We might be good here now. The latest build stopped at the push stage:

2023-10-12T12:25:00.8937972Z ERROR: failed to solve: failed to push techknowlogick/xgo:toolchain: push access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

with the toolchain image building successfully.

WRT to packages: the major changes I made are:

• remove gcc-9-multilib and gcc-9-aarch64-linux-gnu
• add gcc-9-arm-linux-gnueabihf-base and gcc-9-arm-linux-gnueabihf

I am not sure what gcc-9-multilib really does or if its needed - and if so, it issues would only appear at runtime? Do you more here?

Also I am not if gcc-9-arm-linux-gnueabihf are effective without additional setup as https://askubuntu.com/questions/250696/how-to-cross-compile-for-arm/250721#250721 suggests more changes when they should actually be used.

If you think it might work, I can clean up here again to get it ready for merge.

[pat-s]

🎉 Finally got it working - this time for real

Here's proof: https://ci.woodpecker-ci.org/repos/3780/pipeline/8876/10

An amd64 runner using the changed arm64 gcc compiler (which is also available for Ubuntu arm) to cross-compile.

Key changes are:

• Use gcc-arm-linux-gnueabi and binutils-arm-linux-gnueabi as arm compiler instead of aarch64-linux-gnu-gcc-9
• Remove gcc-9-multilib (seems unneeded and is not avail for Ubuntu arm)

I've cleaned up the PR. While it won't succeed due to push errors to Dockerhub, it should build when merging to main.

If you're unsure if everything is good you can also build distinct images of course. In https://hub.docker.com/repository/docker/pats22/xgo I've pushed some multiarch images for testing that were also used for the above linked WP build.

[6543]

CI fail ?

[techknowlogick]

@6543 I'm having trouble building this locally. I'm getting "patchmake" is missing, but I see it on the Ubuntu package site. Will need to dive in a bit more.

[pat-s]

I think patchmake is not needed. Unclear why I added it at some point...(I could reproduce the issue). I've removed it again and the toolchain seems to build fine now (again).

Here's are some local helper commands I used for building, maybe they are of help:

### FIRST (TOOLCHAIN)
cd docker/toolchain
docker buildx build -t pats22/xgo:toolchain -f Dockerfile --platform linux/arm64 --load .
cd ../..
# publish
cd docker/toolchain
docker buildx build -t pats22/xgo:toolchain -f Dockerfile --platform linux/amd64,linux/arm64 --push .
cd ../..

### SECOND (BASE)
# 1.21.3
# load
docker buildx build -t pats22/xgo:go-1.21.3-base -f docker/go-1.21.3/Dockerfile --platform linux/arm64 --load .
# publish
docker buildx build -t pats22/xgo:go-1.21.3-base -f docker/go-1.21.3/Dockerfile --platform linux/amd64,linux/arm64 --push .

### THIRD (COMBINED)
# load
docker buildx build --build-arg VERSION=pats22/xgo:go-1.21.3 -t pats22/xgo:go-1.21.3 -f docker/build/Dockerfile --platform linux/arm64 --load .
# publish
docker buildx build --build-arg VERSION=pats22/xgo:go-1.21.3 -t pats22/xgo:go-1.21.3 -f docker/build/Dockerfile --platform linux/amd64,linux/arm64 --push .

[techknowlogick]

thanks for your patience during this process @pat-s

I built the toolchain and published a temporary tag (https://hub.docker.com/layers/techknowlogick/xgo/toolchain-arm-test/images/sha256-b25050e82fb4ea32f8da82fb51036341adf0599a189232a334bd64cc4cfdc327?context=explore). Now I now need to build the specific version image, and test building some binaries.

[techknowlogick]

Now I've built the specific tag https://hub.docker.com/layers/techknowlogick/xgo/arm-1.21.4/images/sha256-ad9f0e5133ca5338d7e9a63d5697ae321a09969f8fe2b29d073d808c6d842429?context=explore for 1.21.4 Next is to build a binary and test it.

[pat-s]

We can do the testing in WP which is using xgo to cross-compile binaries. That was the initial motivation anyhow 🙂

Spoiler: with my own images this already worked fine

-> https://github.com/woodpecker-ci/woodpecker/pull/2605

I see an error

# go.woodpecker-ci.org/woodpecker/cmd/server
/usr/bin/ld: /tmp/go-link-3311643038/000012.o: in function `unixDlOpen':
/woodpecker/src/github.com/woodpecker-ci/woodpecker/vendor/github.com/mattn/go-sqlite3/sqlite3-binding.c:44399: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Cleaning up build environment...
mkdir -p ./dist/server/linux/amd[64](https://ci.woodpecker-ci.org/repos/3780/pipeline/9628/9#L64)
mv /build/woodpecker-server-linux*-amd64 ./dist/server/linux/amd64/woodpecker-server
[ "" -eq "1" ] && tar -cvzf dist/woodpecker-server_linux_amd64.tar.gz -C dist/server/linux/amd64 woodpecker-server || echo "skip tar.gz binary"
skip tar.gz binary
/bin/sh: 1: [: Illegal number: 
make[1]: Leaving directory '/woodpecker/src/github.com/woodpecker-ci/woodpecker'
tree dist
dist
`-- server
    `-- linux
        `-- amd64
            `-- woodpecker-server

3 directories, 1 file

Didn't spot this one before. It also appeared in run which used my image. Need to investigate...

[techknowlogick]

I'm also getting:

Enabled Go module support
Using vendored Go module dependencies
Building /source/go.mod...
Compiling for linux/amd64...
# runtime/cgo
gcc: error: unrecognized command line option '-m64'

when building for amd64 on arm (need to test the reverse still).

[pat-s]

I remember that during the image build there were some tests which tested the functionality across different architecturs? AFAIR they succeeded so I am wondering where this now comes from 🥴

[6543]

@techknowlogick :wave: :)

[pat-s]

@6543 there is still an issue with the gcc arm compiler last time I tried it :( I hadn't yet time or an idea how to resolve it. No action required by tech

[6543]

uh ok :( !

[pat-s]

OK, some good news: I did another deep dive with a lot of coffee as I can't stand not solving this puzzle 🤪️

Findings:

• a valid compiler for amd64 for both amd64 and arm64 hosts is x86_64-linux-gnu-gcc. Unfortunately, this one doesn't come in one package for amd64 and arm64 on Ubuntu but is bundled in gcc-multilib-x86-64-linux-gnu (arm) and `
• a valid compiler for arm64 for both amd64 and arm64 hosts is arm-linux-gnueabihf-gcc. This one is available for both archs in gcc-arm-linux-gnueabihf.
• Ubuntu arm64 doesn't have packages for gcc-mips*, hence cross-compiling for this arch on arm64 likely doesn't work. I did not test it though. Maybe there is also another package with a different name providing this toolchain.
• so far, no CC and CXX were defined for amd64 in build.sh as it was the "home arch". After adding arm64 as an alternative "home arch" this is now needed and added.

I tested the following scenarios on the example of cross-compiling woodpecker-server:

• Host amd64 -> compile for amd64
• Host amd64 -> compile for arm64
• Host arm64 -> compile for arm64
• Host arm64 -> compile for amd64

docker run --pull=always --rm -it pats22/xgo:go-1.21.4 bash

git clone --depth 1 https://github.com/woodpecker-ci/woodpecker.git
cd woodpecker/

curl -fsSL https://deb.nodesource.com/setup_lts.x | bash -
apt install -y nodejs
curl -fsSL https://get.pnpm.io/install.sh | bash -
source /root/.bashrc

make build-ui

TARGETOS=linux TARGETARCH_XGO=amd64 TARGETARCH_BUILDX=linux/amd64 make release-server-xgo
rm -rf /deps
TARGETOS=linux TARGETARCH_XGO=arm64 TARGETARCH_BUILDX=linux/arm64 make release-server-xgo

I also ran a build using the image above directly in Woodpeckers CI (using the amd64 image): https://ci.woodpecker-ci.org/repos/3780/pipeline/11320/9. In a PR this only cross-compiles amd64 -> amd64 but this one also succeeded!

Will now rebuild images for this PR and then @techknowlogick it might be worth testing again :)

[6543]

did a test run at https://ci.woodpecker-ci.org/repos/3780/pipeline/11566/2

[pat-s]

@techknowlogick The test run (linked by 6543) went fine WRT to xgo (the one failing step is an unrelated failure). Waiting for your feedback 🙂

[pat-s]

ping @techknowlogick

[6543]

integration test now fails ... loksl like the testdata was not pinned - unrelated?

[pat-s]

CI failures look unrelated. Also the toolchain itself builds fine.

# Fetching main repository github.com/rwcarlsen/cyan/cmd/cyan...
# go: modules disabled by GO111MODULE=off; see 'go help modules'
# /build.sh: line 114: cd: /go/src/github.com/rwcarlsen/cyan/cmd/cyan: No such file or directory

[...]

# go: downloading github.com/go-playground/locales v0.14.1
# frontend/embed.go:21:12: pattern dist: no matching files found
# Cleaning up build environment...
# chown: cannot access '/build/code.vikunja.io/api*': No such file or directory
# 2024/04/10 07:54:27 Failed to cross compile package: exit status 1.

[pat-s]

@techknowlogick FYI the image has been in use lately in WP itself and all cross-compile steps worked: https://ci.woodpecker-ci.org/repos/3780/pipeline/15613/9

Great to see this getting merged! Let me know if any issues occur.

[6543]

thanks for the work !!!

[techknowlogick]

@pat-s thanks for your work on this PR and for your patience too.