search

cross-rs / cross

“Zero setup” cross compilation and “cross testing” of Rust crates
Apache License 2.0
6.21k stars 354 forks source link

Tracking Issue for Aarch64 Images #975

Open Alexhuszagh opened 1 year ago

Alexhuszagh commented 1 year ago

This is a tracking issue for building aarch64 images, meaning they run natively on linux/arm64 instead of linux/amd64, and what's confirmed to work, what needs to be fixed, and what's currently untested, so we can ensure the maintainers don't duplicate work, new contributors know where to start, and looking at what to patch for future builds. We can also likely add builds for PPC64LE, since almost all of the maintained packages that have install candidates for ARM64 also support PPC64LE. Currently, most of the linux GNU packages are/will be failing.

Note that the PPC64LE images are currently untested, but assumed to work.

Currently Succeeding

Builds of these images for aarch64 can be found here for debugging and testing purposes (these are not official images, not are they guaranteed to be stable).

Currently Failing

Failing, Cannot Fix

The Android images have the following requirements, which means it will be impossible to build a cross-compiler for them without significant effort from a non-x86 host:

  1. They require gcc-multilib and g++-multilib, which are not available on aarch64 and any host other than i686, x86_64, and s390x.
  2. They use prebuilts for clang, gcc and go that assume an x86 host.
  3. The NDKs assume an x86_64 host.

The only solution therefore would be to rebuild the entire NDK from source, get Android system, replace the prebuilts with native toolchains for a non-x86 host, update code so it uses the correct non-x86 prebuilts, etc. If this sounds like an issue, it is.

Currently Untested

Contributing

If you'd like to contribute your own tests, successes, etc., feel free to do so with the following examples. The runners, std, C++, and dynamic library support can be found in the Github actions ci.yml.

# change this for specific targets
$ export TARGET=s390x-unknown-linux-gnu
$ cargo build-docker-image --platform linux/arm64/v8=aarch64-unknown-linux-gnu $TARGET
$ export CROSS_TARGET_S390X_UNKNOWN_LINUX_GNU_IMAGE_TOOLCHAIN="aarch64-unknown-linux-gnu"
$ export CROSS_TARGET_S390X_UNKNOWN_LINUX_GNU_IMAGE="ghcr.io/cross-rs/$TARGET:local"
$ export STD=1
$ export CPP=1
$ export DYLIB=1
$ export RUN=1
$ export RUNNERS="qemu-user qemu-system"
$ ci/test.sh
Alexhuszagh commented 1 year ago

If we install packages from Debian for some of these toolchains, we can support a lot more architectures. Potential solutions include:

This only seems to affect MIPS however.

DoumanAsh commented 1 year ago

Just observation I encountered issue with latest libc crate (0.2.126) requiring newer glib version

  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)
  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)
  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)

It would be good to consider bumping ubuntu images as they have relatively old glib for cross compilation

Alexhuszagh commented 1 year ago

Just observation I encountered issue with latest libc crate (0.2.126) requiring newer glib version

  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)
  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)
  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)

It would be good to consider bumping ubuntu images as they have relatively old glib for cross compilation

That's unrelated (and an upstream bug): see #724. Try running cargo clean and then rebuilding. This can occur in any repository with a build script. It can happen fairly regularly if switching between toolchains. This is a cargo bug, but is exacerbated by how cross functions (swapping toolchains with the same target directory, which causes the bug to manifest).

Emilgardis commented 1 year ago

@CoreyCole this is the wrong issue for your problem, please create a new issue

as a quick mention, the table in the readme is only for the current :main images, you're using :latest in your example. Try 

[target.aarch64-unknown-linux-gnu]
image = "ghcr.io/cross-rs/aarch64-unknown-linux-gnu:main"
abcd-ca commented 1 year ago

I downloaded an aarch64 image to my host (my M1 mac). Would someone tell me how to use it? I've got a knowledge gap between having the native (M1, aarch64) image and this command: CROSS_REMOTE=1 DOCKER_HOST=tcp://docker:2375/ cross build --release --target=armv7-unknown-linux-musleabihf.

I think for me the DOCKER_HOST part is wrong and when I try to run the image (start a container based on it) I see no port has been exposed. image

Thank you

Alexhuszagh commented 1 year ago

@abcd-ca

Do you need CROSS_REMOTE? You should be able to run it natively on your Mac M1, and therefore don't need remote docker, unless you are connecting to the client remotely. You do have to provide the image name itself, however: these aren't named the same so you have to override the image name as follows:

CROSS_TARGET_ARMV7_UNKNOWN_LINUX_MUSLEABIHF_IMAGE=ahuszagh/aarch64-cross:armv7-unknown-linux-musleabihf \
    cross build --release --target=armv7-unknown-linux-musleabihf

Does that help? All the image names and tags can be found here.

abcd-ca commented 1 year ago

@Alexhuszagh Thanks, that does look more like what I need however with that I get,

docker: Error response from daemon: image with reference ahuszagh/aarch64-cross:armv7-unknown-linux-musleabihf was found but does not match the specified platform: wanted linux/amd64, actual: linux/arm64.

Alexhuszagh commented 1 year ago

Is your local machine the M1? Or are you connecting remotely to it? If so, then you do need CROSS_REMOTE along with the correct connection parameters (SSH connection? TCP?).

abcd-ca commented 1 year ago

Local is M1. No physically remote build computer yet. Hoping to avoid that

Alexhuszagh commented 1 year ago

Local is M1. No physically remote build computer yet. Hoping to avoid that

Ah I think I know what the issue is. Sorry for the repeated mentions. You need to do the instructions under target.TARGET.image, or provide them as environment variables.

So the full invocation is:

CROSS_TARGET_ARMV7_UNKNOWN_LINUX_MUSLEABIHF_IMAGE_TOOLCHAIN="aarch64-unknown-linux-gnu" \
    CROSS_TARGET_ARMV7_UNKNOWN_LINUX_MUSLEABIHF_IMAGE=ahuszagh/aarch64-cross:armv7-unknown-linux-musleabihf \
    cross build --release --target=armv7-unknown-linux-musleabihf

If we deploy native ARM64 images, this should be streamlined a lot later on (we need custom runners which aren't free), but right now it's a bit janky. Does that fix the issue?

abcd-ca commented 1 year ago

Sorry for the repeated mentions

Are you kidding? Thanks for being so generous with your time and expertise!

Success! 🎉

Here are some test results (docker images already automatically downloaded so time to download those not included):

Build time with default build image cross clean time ./build.sh

Finished release [optimized] target(s) in 8m 45s ./build.sh 0.46s user 0.26s system 0% cpu 8:48.76 total

Build time with native aarch64 (M1) build image: cross clean time ./build.sh (modified to use your latest env vars)

Finished release [optimized] target(s) in 2m 23s ./build.sh 0.65s user 0.64s system 0% cpu 2:27.52 total

Summary:

About 6.5 minutes (73%) faster!

jaysonsantos commented 1 year ago

Hey there folks, are these images being pushed somewhere? In this repo, I can only see linux/amd64 and unknown/unknown

image
Emilgardis commented 1 year ago

@Alexhuszagh has pushed initial images here as mentioned in this issue.

We have not started implementing this in-org yet, and thus they are not on ghcr.io/cross-rs. You can also build these images yourself

YOU54F commented 1 year ago

Hey @Alexhuszagh

Kudos on the awesome tracking list, I have this saved as a bookmark, was SUPER useful for me, in my current ARMaggedon flex on our open source project.

Linked PR above ☝🏾 to attempt to cover off the case for aarch64-unknown-freebsd

DoumanAsh commented 10 months ago

Did someone break mips64-unknown-linux-gnuabi64 target image?

Recently I started seeing failures when cross compiling:

Digest: sha256:de63e12792fb659b1f7fdbe0bfbee8d45cc25ecd39a1c22803a5ad31[74](https://github.com/DoumanAsh/xxhash-rust/actions/runs/5965359760/job/16182657559#step:5:75)a8524c
Status: Downloaded newer image for ghcr.io/cross-rs/mips64-unknown-linux-gnuabi64:0.2.5
   Compiling libc v0.2.147
   Compiling cfg-if v1.0.0
error[E0463]: can't find crate for `core`
  |
  = note: the `mips64-unknown-linux-gnuabi64` target may not be installed
  = help: consider downloading the target with `rustup target add mips64-unknown-linux-gnuabi64`

error[E0463]: can't find crate for `compiler_builtins`
CryZe commented 10 months ago

That could be related to 1.72 that just released. I recall that mips got downgraded a tier, though I don't see it in the release notes.

DoumanAsh commented 10 months ago

Hm downgraded? According to https://doc.rust-lang.org/nightly/rustc/platform-support.html mips64-unknown-linux-gnuabi64 is Tier 2 so it should support build with standard library.

idk though, it could be easily broken since people are unlikely to care much about this target

CryZe commented 10 months ago

It seems like the docs have not been updated as well as the release notes? https://github.com/rust-lang/rust/pull/113274

Emilgardis commented 10 months ago

I fail to see how this is related to this specific tracking issue?

mcandre commented 9 months ago

By the way, Rust added support for a NetBSD aarch64 big endian target:

https://github.com/rust-lang/rust/pull/111326/

Emilgardis commented 9 months ago

By the way, Rust added support for a NetBSD aarch64 big endian target:

rust-lang/rust#111326

Cool! However this issue is specifically for aarch host images!

mcandre commented 6 months ago

Configuring a DOCKER_DEFAULT_PLATFORM environment variable with value linux/amd64 can smooth out rough edges while awaiting publication of multi-architecture images. You may need to explicitly initialize the Docker buildx subsystem, something that the tug third party utility can do for ya.

https://github.com/mcandre/tug

Recommend to house this configuration in direnv / psenv, in order to reduce conflicts between different software projects you may be working on.