What is the state of CMLFS methods for cross-compiling?

[takusuman]

Recently or, to be more specific, yesterday, I decided to make a build of Copacabana with LLVM/clang, and I would like to know if this project already runs on ARM and i{4,5,6}86. I'm inclined to cooperate if needed, like I did in 2021 at the original Musl-LFS.

[dslm4515]

I am successful on x86_64. I'm revising the LLVM 15.0.6 build/branch for the last time before upgrading to 16.0.6.

I've attempted a 32bit build (i686) but hit a snag while compiling python3 for the final system: built python locks up the the system and will not finish the build...even if I left it over night. Cgnutools and llvmtools work as intended.... tricky as LLVM seems to use i386 built is for i686... kind of like Musl Libc.

With the release of Raspberry Pi, I'm getting my pi4 ready to build a aarch64 build.

[takusuman]

I've attempted a 32bit build (i686) but hit a snag while compiling python3 for the final system: built python locks up the the system and will not finish the build...even if I left it over night.

What is your CPU? This sounds more like a CPU bug to me. Already tried to compile it in a virtual i386? Like 86Box or even QEmu.

With the release of Raspberry Pi, I'm getting my pi4 ready to build a aarch64 build.

Well, have you already got anything from ARM/aarch64? Just out of curiosity.

Another question: how is the performance of this? Is it really worse than GCC built Musl-LFS as some LLVM haters say? And how could I measure it by myself?

[dslm4515]

What is your CPU?

Intel Core Duo T2600 in an old HP Compaq nc6400. I then tried resuming the build on an old mini-iTX board that has an Intel Atom N2800...but ran the chroot in 32-but mode. Same issue. Build hangs on ./python -E ./setup.py build.

[takusuman]

What is your CPU?

Intel Core Duo T2600 in an old HP Compaq nc6400. I then tried resuming the build on an old mini-iTX board that has an Intel Atom N2800...but ran the chroot in 32-but mode. Same issue. Build hangs on ./python -E ./setup.py build.

That's really strange, maybe it's not your CPU fault. Maybe we should try contacting the Python development team?

[dslm4515]

My amd64 build is 36 packages behind my i686 build... I will see if there was a mistake in the i686 build or if the same issue shows up in the amd64 build.

I have found my RPi4 but not sure if it can still boot from the USB(3.0)-SATA 2.5" SSD it has.

[dslm4515]

Finished python3 for the final system in my amd64 build: No issues. Looks like there is a problem with my 32-bit build OR I need to make adjustments for 32-bit x86.

[takusuman]

Looks like there is a problem with my 32-bit build OR I need to make adjustments for 32-bit x86.

I wish I had a i{3,4,5,6}86 hanging around so I could test this out, but for now I'm only going with x86_64 for Copacabana. May I try building it via 86Box later just for fun and see if I can find something out.

[dslm4515]

Per CLFS, you could build i686 on a x86_64 machine via 'setarch' or a kernel module:

If your host has a tool called setarch, this may solve your problems, at least if you're building for i686. On an architecture such as x86_64, using "setarch linux32 uname -m " will only ever output i686. It is not possible to get an output of i486 or i586.

or even "Uname Hack":

The Uname Hack is a kernel module that modifies the output of uname -m by directly changing the value of the detected machine type. The kernel module will save the original value and restore it when the module is unloaded.

[dslm4515]

I built llvmtools for i686 on a amd64 machine then used setarch with chroot to enter a "32-bit" chroot environment.

[dslm4515]

I have completed an i686 build on an x86_64 host. Biggest issue was stage 2 python3 compiles but stalls with any python script like pip. Workaround for it was building stage 2 python with stage1 clang instead of stage 2.

I am now doing another i686 build on another x86_64 to clean up the repo and verify it works.

For "non-similar" cross builds, like arm64 on a x86_64 host most likely will need a different build method. I probably will create another repo like ... xCMLFS?

[takusuman]

May I try building it via 86Box later just for fun and see if I can find something out.

Pardon for not giving any return yet, I have been a little bit busy on other projects beside Copacabana --- but on which it depends --- and I couldn't test any of the hypothesis that you made yet.

[dslm4515]

No problem! My MLFS repo is getting stale as the last few months have I havetaken a break (built some plastic models, gunpla, in my free time instead of updating my repos) and started working on updating CMLFS.

[takusuman]

For "non-similar" cross builds, like arm64 on a x86_64 host most likely will need a different build method. I probably will create another repo like ... xCMLFS?

Why? How much would it differ from building from one platform to itself?

[dslm4515]

Say, I want to build CMLFS for a Raspberry Pi 4 on my x86_64 host.

I would have to build a cross-toolchain (cgnutools) that runs on x86_64 but produces binaries that run on aarch64. Of course, a binary built for aarch64 cannot run natively on a x86_64 CPU. I would have to run some sort of emulation on my x86_64 host to run aarch64 code. _This is not needed for i686/x86_64 as x8664 CPUs can run 32-bit code (usually most Linux kernels have support to run 32-bit code on 64-bit x86 CPUs).

Then cgnutools would be used to build llvmtools that runs on aarch64 and produces binaries for aarch64. Again, I will need some sort of emulation to run llvmtools on my x86_64 host.

Therefore for "non-similar" cross-builds, I would have to deploy some sort of emulation, such as QEMU (run a virtual machine) or binfmt-support.

There is Cross-LFS, but it relies on booting a minimal LFS system on hardware that supports the target CPU architecture:

Booting is required when you are compiling for a different architecture, such as building a PowerPC system from an x86. The rule of thumb here is that if the architectures match and you are running the same series kernel (specifically, a 2.6.32 or newer Linux kernel) you can just chroot. If you aren't running the same series kernel, or are wanting to run a different ABI, you will need to use the boot option.

[takusuman]

I would have to build a cross-toolchain (cgnutools) that runs on x86_64 but produces binaries that run on aarch64. Of course, a binary built for aarch64 cannot run natively on a x86_64 CPU. I would have to run some sort of emulation on my x86_64 host to run aarch64 code.

Oh, I was aware of this, but can we cross-build it all instead of breaking it in two stages or the way a Linux system is built forbids us? Like was done on SunOS, for instance, where it was entirely cross-built from one platform for another if needed.

Again, I will need some sort of emulation to run llvmtools on my x86_64 host.

QEmu could be a good shot, but, correct me if I'm erred, maybe an ARM-specific emulator can perform better for cross-compiling an entire system than QEmu.

I'll be studying some of this since I'm with some free time lately.

[dslm4515]

but can we cross-build it all instead of breaking it in two stages

That's a good point. I would think it might be possible to cross-compile each package... but I'm not sure if some packages require running a binary that runs on the target.

Since I managed to build a "skeleton" PC (unfinished case, bare minimal parts) with 6 cores, I can try cross-compiling aarch64 on it.

[takusuman]

but I'm not sure if some packages require running a binary that runs on the target

Sure thing that the biggest problem on making something such this is the fact that we don't have control of all the components on the system, in the sense of how it work. If you manage to do it, I would be happy to know. I'm currently writing a build system for Copacabana and cross-compiling it entirely without needing to boot QEmu and/or enter in chroot would help.

[dslm4515]

So I have begun cross-compiling aarch64 on x86_64.

As expected, mussel script executed fine, bootstrapping the cgnutools. My first obstacle is cross-compiling base LLVM. Configuring works, but the build fails:

[204/3805] Building GenVT.inc...
FAILED: include/llvm/CodeGen/GenVT.inc /mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build0/include/llvm/CodeGen/GenVT.inc 
cd /mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build0 && /mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build0/bin/llvm-min-tblgen -gen-vt -I /mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/llvm/include/llvm/CodeGen -I/mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build0/include -I/mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/llvm/include /mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/llvm/include/llvm/CodeGen/ValueTypes.td --write-if-changed -o include/llvm/CodeGen/GenVT.inc -d include/llvm/CodeGen/GenVT.inc.d
/mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build0/bin/llvm-min-tblgen: 1: Syntax error: "(" unexpected
[215/3805] Building CXX object lib/DebugInfo/CodeView/CMakeFiles/LLVMDebugInfoCodeView.dir/EnumTables.cpp.o
ninja: build stopped: subcommand failed.

Not sure if I need to patch LLVM source more.

[dslm4515]

Did a quick Google search. Perhaps I need to build llvm-tblgen and clang-tblgen for host first?

I first had to disable cgnutools and llvmtools as my host's GCC tried to use the cross-compiled libraries in cgnutools... built just llvm-tblgen and clang-tblgen in a seperate build folder, build-host.

Then I appended the cmake invocation with:

-DLLVM_TABLEGEN=/mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build-host/bin/llvm-tblgen 
-DCLANG_TABLEGEN=/mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build-host/bin/clang-tblgen

But the build still fails:

Error relocating /cgnutools/aarch64-pc-linux-musl/lib/libgcc_s.so.1: unsupported relocation type 1025

I suppose aarch64 libraries are being loaded during the build? If so, then I would need to deploy some emulation for aarch64 on x86_64.

[takusuman]

Syntax error: "(" unexpected

That sounded like a undefined variable when auto-generating a file, strange.

[takusuman]

I suppose aarch64 libraries are being loaded during the build? If so, then I would need to deploy some emulation for aarch64 on x86_64.

So we're now back to ground one? Hmmm... I'm a little bit busy making a test suite for Heirloom NG, but I will try to research this some way later. There's no way something like this is impossible, principally on Linux.

[dslm4515]

So we're now back to ground one?

Yes, somewhat. I am now trying your suggestion of building a cross-compiler from LLVM instead of GCC (via mussel)... this means ditching mussel and using the host's GCC to build stage 0 LLVM with cross-compiling support.

[takusuman]

this means ditching mussel and using the host's GCC to build stage 0 LLVM with cross-compiling support.

Correct me if I'm erred, but does not mussel support building a cross-compiler for another architectures? I had not studied it enough yet, so pardon for my unawareness. Maybe if we get out of this with a solution it can be applied into mussel itself instead of ditching it. I do not know a lot about cross-compiling yet so, if you can and don't mind to, link some references about it, please.

[dslm4515]

My fault, I left out a few details. I did initially start with mussel but stage0 clang kept using the runtimes that mussel built.

But yes, mussel does support cross-compiling. Perhaps I am wasting time trying to build a cross-compiling stage 0 LLVM without mussel. I may need to use mussel to build a cross-gcc and disable it when running stage0 clang.

[dslm4515]

Stage0 clang kept using the runtimes that mussel built.

When I tested this stage 0 clang, it would compile for the target (aarch64) but then link against runtime obejcts of the host:

/cgnutools/bin/aarch64-pc-linux-musl-ld" -pie -EL -z now -z relro --hash-style=gnu --eh-frame-hdr -m aarch64linux -dynamic-linker /llvmtools/lib/ld-musl-aarch64.so.1 -o a.out /lib/Scrt1.o /lib/crti.o /cgnutools/bin/../lib/gcc/aarch64-pc-linux-musl/13.2.0/crtbeginS.o -L/cgnutools/bin/../lib/gcc/aarch64-pc-linux-musl/13.2.0 -L/lib/ -L/usr/lib/ -L/cgnutools/bin/../lib/gcc/aarch64-pc-linux-musl/13.2.0/../../../../aarch64-pc-linux-musl/lib -L/lib -L/usr/lib --as-needed /tmp/dummy-85793f.o --verbose -lgcc --as-needed -lgcc_s --no-as-needed --push-state --as-needed -latomic --pop-state -lc -lgcc --as-needed -lgcc_s --no-as-needed /cgnutools/bin/../lib/gcc/aarch64-pc-linux-musl/13.2.0/crtendS.o /lib/crtn.o

/cgnutools/bin/aarch64-pc-linux-musl-ld: mode aarch64linux
attempt to open /lib/Scrt1.o succeeded
/lib/Scrt1.o
/cgnutools/bin/aarch64-pc-linux-musl-ld: /lib/Scrt1.o: Relocations in generic ELF (EM: 62)
/cgnutools/bin/aarch64-pc-linux-musl-ld: /lib/Scrt1.o: Relocations in generic ELF (EM: 62)
/cgnutools/bin/aarch64-pc-linux-musl-ld: /lib/Scrt1.o: error adding symbols: file in wrong format

Maybe I just need to reconfigure stage clang.

[dslm4515]

Yup. Fixed it with a clang config:

cat > /cgnutools/bin/${TARGET_TUPLE}.cfg <<EOF
-fuse-ld=lld
--sysroot=/cgnutools
EOF

test it via echo "int main(){}" > dummy.c && clang dummy.c -v -Wl,--verbose &> dummy.log Now it loads the correct libraries & runtime objects!

ld.lld: /cgnutools/lib/Scrt1.o
ld.lld: /cgnutools/lib/crti.o
ld.lld: /cgnutools/lib/gcc/aarch64-pc-linux-musl/13.2.0/crtbeginS.o
ld.lld: /tmp/dummy-37efb9.o
ld.lld: /cgnutools/lib/gcc/aarch64-pc-linux-musl/13.2.0/libgcc.a
ld.lld: /cgnutools/lib/gcc/aarch64-pc-linux-musl/13.2.0/../../../../aarch64-pc-linux-musl/lib/libgcc_s.so
ld.lld: /cgnutools/lib/gcc/aarch64-pc-linux-musl/13.2.0/../../../../aarch64-pc-linux-musl/lib/libgcc_s.so.1
ld.lld: /cgnutools/lib/gcc/aarch64-pc-linux-musl/13.2.0/libgcc.a
ld.lld: /cgnutools/lib/libatomic.so
ld.lld: /cgnutools/lib/libc.so
ld.lld: /cgnutools/lib/gcc/aarch64-pc-linux-musl/13.2.0/libgcc.a
ld.lld: /cgnutools/lib/gcc/aarch64-pc-linux-musl/13.2.0/../../../../aarch64-pc-linux-musl/lib/libgcc_s.so
ld.lld: /cgnutools/lib/gcc/aarch64-pc-linux-musl/13.2.0/../../../../aarch64-pc-linux-musl/lib/libgcc_s.so.1
ld.lld: /cgnutools/lib/gcc/aarch64-pc-linux-musl/13.2.0/libgcc.a
ld.lld: /cgnutools/lib/gcc/aarch64-pc-linux-musl/13.2.0/crtendS.o
ld.lld: /cgnutools/lib/crtn.o

[dslm4515]

I do not know a lot about cross-compiling yet so, if you can and don't mind to, link some references about it, please.

Good question. I think I printed out some websites on cross-compiling back in 2017. It helped me under stand the difference between target, build, and host. I'll have to dig that out. I used to help me understand how to "cross-compile" from glibc to musl for my MLFS repo.

I did recently looked at this question asked at Stack Overflow

[takusuman]

Now it loads the correct libraries & runtime objects!

That means that a working cross-compiler was built or that a aarch64 binary could be cross-compiled? Anyway, I remembered about oasis Linux, it cross-compiles from even OpenBSD and from different architectures if I'm not erred. I know that the purpose of it is different from Musl-LFS/CMLFS (and any distribution aimed for desktop use) since it is more of an experiment, but maybe we can learn something off it.

[dslm4515]

It seems so. I was able to cross-compile stage 0 clang with cross-GCC (built by mussel). Stage 0 clang can build aarch64 binaries like chimera's libatomic or even zlib-ng.

BUT! Now build fails for stage 2 clang, using stage 0 clang. The tblgen binaries fail to build (unresolved symbols).

[takusuman]

How does Chimera, Alpine and other Linux distributions do automatic nightly (and even release) builds? I highly doubt that the team maintains a QEmu (even if headless and fine-tuned on scripts) "just" to cross-build. I'm thinking about doing some study on it later, but I think my idea may help CMLFS before Copacabana.

[dslm4515]

I got stage 1 LLVM to build further, but looks like stage 1 clang fails to build unlike stage 1 lld:

 ninja: Entering directory `build1'
[1/47] ASTNodeAPI.json
FAILED: tools/clang/lib/Tooling/ASTNodeAPI.json /mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build1/tools/clang/lib/Tooling/ASTNodeAPI.json 
cd /mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build1/tools/clang/lib/Tooling && /mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build1/bin/clang-ast-dump --skip-processing=0 -I /mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build1/lib/clang/17/include -I /mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/clang/include -I /mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build1/tools/clang/include -I /mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build1/include -I /mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/llvm/include -I /llvmtools/include/c++/v1 -I /llvmtools/include -I /cgnutools/lib/clang/17/include --json-output-path /mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build1/tools/clang/lib/Tooling/ASTNodeAPI.json
/mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build1/bin/clang-ast-dump: 1: ELF@H@8: not found
/mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build1/bin/clang-ast-dump: 1: 
                                                                                          : not found
/mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build1/bin/clang-ast-dump: 1: 
                                                                                          : not found
PPP..: not foundlfs-aarch64/sources/llvm-project-17.0.6.src/build1/bin/clang-ast-dump: 1: 
/mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build1/bin/clang-ast-dump: 1: : not found
/mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build1/bin/clang-ast-dump: 2: cannot open DKR[biv{: No such file
/mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build1/bin/clang-ast-dump: 2: K: not found
/mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build1/bin/clang-ast-dump: 2:]: not found
/mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build1/bin/clang-ast-dump: 3:  not found
/mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build1/bin/clang-ast-dump: 4: : not found
/mnt/32satam2/cmlfs-aarch64/sources/llvm-project-17.0.6.src/build1/bin/clang-ast-dump: 5: Syntax error: ")" unexpected
[2/47] Linking CXX static library lib/libclangFormat.a
ninja: build stopped: subcommand failed.

I checked that binary to see if needs the rpath set:

$ /cgnutools/aarch64-pc-linux-musl/bin/readelf -d /mnt/cmlfs/sources/llvm-project-17.0.6.src/build1/bin/clang-ast-dump

 0x0000000000000001 (NEEDED)             Shared library: [libc.so]
 0x0000000000000001 (NEEDED)             Shared library: [libc++.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libLLVM-17.so]
...

Make me wonder if build failed because just built clang-ast-dump was an aarch64 binary executed on my x86_64 host

[dslm4515]

How does Chimera, Alpine and other Linux distributions do automatic nightly (and even release) builds?

I'm not sure. I haven't thoroughly looked through their repos to see how things are built. Chimera uses python scripting to automate the build process, I believe.

[dslm4515]

I'm going to take a break from cross-compiling CMLFS.

I probably will have to look into using binfmt & qemu... Maybe use mussel to build a cross-gcc toolchain to cross-compile enough packages to run an "emulated" chroot environment (use qemu to allow aarch64 binaries to run the host CPU while in chroot).

[dslm4515]

I just remembered I tried using crosstool-ng and buildroot back in 2010 when trying to cross-compile LFS for my Acer A200 tablet (armv7).... Perhaps, I should look at those and see if I can use them to cross-compile CMLFS

UPDATE: Perhaps, might be easier to use buildroot to build a "chroot" environment with qemu