search

richfelker / musl-cross-make

Simple makefile-based build for musl cross compiler
MIT License
1.3k stars 266 forks source link

cross building from x86_64 #164

Open Ludea opened 2 years ago

Ludea commented 2 years ago

Hi,

I try to build musl toolchain from an x86_64 host to an aarch64 hosts.

My config.mak:

TARGET = aarch64-linux-musl
OUTPUT = /usr/local
BINUTILS_VER = 2.38
GCC_VER = 12.1.0
LINUX_VER = 5.15.44
SHA1_CMD = sha1sum -c
COMMON_CONFIG += CC="aarch64-linux-musl-gcc -static --static" CXX="aarch64-linux-musl-g++ -static --static"
COMMON_CONFIG += CFLAGS="-g0 -Os" CXXFLAGS="-g0 -Os" LDFLAGS="-s"
COMMON_CONFIG += --disable-nls
GCC_CONFIG += --disable-libquadmath --disable-decimal-float
GCC_CONFIG += --disable-libitm
GCC_CONFIG += --disable-fixed-point
GCC_CONFIG += --disable-lto

I get: configure: error: in `/home/runner/work/musl-cross-make/build/local/aarch64-linux-musl/obj_binutils': configure: error: C compiler cannot create executables If I delete COMMON_CONFIG += CC="aarch64-linux-musl-gcc -static --static" CXX="aarch64-linux-musl-g++ -static --static" build is successful.

I miss something ? or build is broken with static flag ?

rofl0r commented 2 years ago

not what you're asking, but:

BINUTILS_VER = 2.38 GCC_VER = 12.1.0

you should only use versions that have an associated directory below patches/ , otherwise the compiler created will have all sorts of issues as it will lack critical compatibility patches for musl.

Ludea commented 2 years ago

I revert to use right version, but still have issue

Some logs : https://github.com/Ludea/speedupdate-rs/actions/runs/3205211691/jobs/5237438973

ryanwoodsmall commented 2 years ago

Still have something broken with your config:

2022-10-07T13:47:41.6264418Z make: *** No rule to make target 'linux-5.15.44', needed by 'all'.  Stop.

Remove the LINUX_VER config setting and go with the default here. You're probably still going to hit a cross-compilation problem, though, as what you're doing appears to be more like a "Canadian cross," i.e.:

x86_64 (running on & building for native arch) -> aarch64 (running on native arch, building for foreign arch) -> aarch64 (running on & building for foreign arch)

There's a long-standing issue with GCC that has been surfaced on (at least) canadian-cross aarch64 builds. A link to an old GCC mention - this exhibits as undefined references to host_detect_local_cpu and I've hit it on GCC 9.x:

The __aarch64__ symbol appears to be defined in the build for that target platform, at least on versions <10, regardless of the host platform, which (I believe) causes native CPU detection to be attempted when running the aarch64 compiler on x86_64, which causes the CPU detection routine to not be defined/found, since it's not compiling native code. I'm not sure if this is still broken on GCC 11 or 12, but IIRC I hit it on the first 10.x release as well. Easiest way to avoid that issue is to patch the middle build (native x86_64 compiler binaries targeting aarch64) with a patch like so - this is for 9.4.0, different versions of GCC will vary:

# cat patches/gcc-9.4.0/1111-no-__aarch64__-symbol.diff.OFF 
--- a/gcc/config/aarch64/aarch64.h      2022-10-07 19:58:21.284795937 +0000
+++ b/gcc/config/aarch64/aarch64.h      2022-10-07 19:58:32.580763886 +0000
@@ -1083,7 +1083,7 @@
 #define MCPU_TO_MARCH_SPEC_FUNCTIONS \
   { "rewrite_mcpu", aarch64_rewrite_mcpu },

-#if defined(__aarch64__)
+#if defined(NOT__aarch64__)
 extern const char *host_detect_local_cpu (int argc, const char **argv);
 #define HAVE_LOCAL_CPU_DETECT
 # define EXTRA_SPEC_FUNCTIONS                                          \

Remove this patch for the final native aarch64 toolchain. If it's not removed for the final round of compilation, I'm not sure what the resulting compiler will do when mtune/march/etc. is set to native/local CPU.

This leads to the next issue: during compilation of the final native aarch64 compiler, the cross-compiler running on x86_64 and targeting aarch64 will possibly (likely?) generate code that needs to run on x86_64 but is compiled for aarch64. Unless qemu-user is installed and working, this will probably fail. It'll be slow even if it works.

Finally, you probably shouldn't mix binaries for different architectures in the same installation directory. While it shouldn't hurt anything, it's asking for trouble.

Here's how I'd go about achieving what you're aiming for here. Note that you will probably need the qemu-user-static package installed and working on Debian or Ubuntu for this...

config.mak snippets for each of the compiler building steps, dropping stuff into /usr/local/musl per-arch - remove ccache if you're not going to be re-running compiles; it significantly speeds up the first (native) pass if you'll run it multiple times, but will slow down same if you're only running it once.

native x86_64 bootstrap musl compiler

# 0_config.mak.bootstrap 
TARGET = x86_64-linux-musl
OUTPUT = /usr/local/musl/bootstrap
DL_CMD = curl -s -C - -L -o
COMMON_CONFIG += CC="ccache gcc -static --static"
COMMON_CONFIG += CXX="ccache g++ -static --static"
COMMON_CONFIG += CFLAGS="-g0 -Os"
COMMON_CONFIG += CXXFLAGS="-g0 -Os"
COMMON_CONFIG += LDFLAGS="-s"
COMMON_CONFIG += --disable-nls
GCC_CONFIG += --disable-libquadmath
GCC_CONFIG += --disable-decimal-float
GCC_CONFIG += --disable-libitm
GCC_CONFIG += --disable-fixed-point
GCC_CONFIG += --disable-lto

native x86_64 musl compiler

# 1_config.mak.native-x86_64 
TARGET = x86_64-linux-musl
OUTPUT = /usr/local/musl/x86_64
DL_CMD = curl -s -C - -L -o
COMMON_CONFIG += CC="ccache /usr/local/musl/bootstrap/bin/x86_64-linux-musl-gcc -static --static"
COMMON_CONFIG += CXX="ccache /usr/local/musl/bootstrap/bin/x86_64-linux-musl-g++ -static --static"
COMMON_CONFIG += CFLAGS="-g0 -Os"
COMMON_CONFIG += CXXFLAGS="-g0 -Os"
COMMON_CONFIG += LDFLAGS="-s"
COMMON_CONFIG += --disable-nls
GCC_CONFIG += --disable-libquadmath
GCC_CONFIG += --disable-decimal-float
GCC_CONFIG += --disable-libitm
GCC_CONFIG += --disable-fixed-point
GCC_CONFIG += --disable-lto

native musl cross compiler targeting aarch64

  1. Install the patch for gating the __aarch64__ symbol here...
# 3_config.mak.cross-aarch64 
TARGET = aarch64-linux-musl
OUTPUT = /usr/local/musl/x86_64
DL_CMD = curl -s -C - -L -o
COMMON_CONFIG += CC="ccache /usr/local/musl/x86_64/bin/x86_64-linux-musl-gcc -static --static"
COMMON_CONFIG += CXX="ccache /usr/local/musl/x86_64/bin/x86_64-linux-musl-g++ -static --static"
COMMON_CONFIG += CFLAGS="-g0 -Os"
COMMON_CONFIG += CXXFLAGS="-g0 -Os"
COMMON_CONFIG += LDFLAGS="-s"
COMMON_CONFIG += --disable-nls
GCC_CONFIG += --disable-libquadmath
GCC_CONFIG += --disable-decimal-float
GCC_CONFIG += --disable-libitm
GCC_CONFIG += --disable-fixed-point
GCC_CONFIG += --disable-lto
  1. Remove the patch for gating the __aarch64__ symbol here.

native aarch64 musl compiler

# 5_config.mak.native-aarch64 
TARGET = aarch64-linux-musl
OUTPUT = /usr/local/musl/aarch64
DL_CMD = curl -s -C - -L -o
COMMON_CONFIG += CC="ccache /usr/local/musl/x86_64/bin/aarch64-linux-musl-gcc -static --static"
COMMON_CONFIG += CXX="ccache /usr/local/musl/x86_64/bin/aarch64-linux-musl-g++ -static --static"
COMMON_CONFIG += CFLAGS="-g0 -Os"
COMMON_CONFIG += CXXFLAGS="-g0 -Os"
COMMON_CONFIG += LDFLAGS="-s"
COMMON_CONFIG += --disable-nls
GCC_CONFIG += --disable-libquadmath
GCC_CONFIG += --disable-decimal-float
GCC_CONFIG += --disable-libitm
GCC_CONFIG += --disable-fixed-point
GCC_CONFIG += --disable-lto

This will drop compilers into their own directories - /usr/local/musl/bootstrap, /usr/local/musl/x86_64 and /usr/local/musl/aarch64. The bootstrap step isn't really necessary, but I'd still recommend doing it just to guarantee you can compile a compiler that can compile itself or something similar.

Note that I can't guarantee that any of this will work, nor can I say the middle aarch64 or resultant aarch64 compiler running on and targeting aarch64 is 100% trustable. For GCC at least, compiling natively, or at least on the same family of CPU (i.e., i686->x86_64, armhf->aarch64, etc.), is still the recommended approach I believe.

zv-io commented 2 years ago

This process is generally much simpler (I do musl.cc things). When I'm back at my computer I'll post some general instructions. The config you are using looks like you need to use my fork of musl-cross-make (the version numbers and compile flags look suspiciously similar).

Ludea commented 2 years ago

This process is generally much simpler (I do musl.cc things). When I'm back at my computer I'll post some general instructions. The config you are using looks like you need to use my fork of musl-cross-make (the version numbers and compile flags look suspiciously similar).

Yes, I started with your fork. I also try to download your binaries with wget, without success, so I try to clone and build both your fork and this repo

zv-io commented 2 years ago

Yes, I started with your fork. I also try to download your binaries with wget, without success, so I try to clone and build both your fork and this repo

Are you trying to download them from inside of GitHub Actions or CI pipelines? See here: https://github.com/orgs/community/discussions/27906

Ludea commented 2 years ago

I try from Github action

Ludea commented 2 years ago

This process is generally much simpler (I do musl.cc things). When I'm back at my computer I'll post some general instructions. The config you are using looks like you need to use my fork of musl-cross-make (the version numbers and compile flags look suspiciously similar).

I'm interresting about some instructions to build it from Ubuntu ;-)

Are you trying to download them from inside of GitHub Actions or CI pipelines? See here: https://github.com/orgs/community/discussions/27906

I fix it, by upload your archive to a static file provider and download the toolchain from it