howto cross-compile to get to work for powerpc e500v2

th0ma7 commented 10 months ago

This relates to SynoCommunity spksrc project to build and package various open source software to run on Synology NAS.

I've been running into an issue where rust code cross-compiled for powerpc arch segfault (more specifically qoriq). The exact same code builds perfectly fine for all other archs (armv5, v7, aarch64, x86_64, i686). Ref: https://github.com/SynoCommunity/spksrc/issues/5847 https://github.com/SynoCommunity/spksrc/issues/5684

I've took two different approaches:

Using RUSTFLAGS to reproduce our CFLAGS used -mcpu=8548 -mhard-float -mfloat-gprs=double https://github.com/SynoCommunity/spksrc/pull/5879/commits/90ad41d43592800a52e8991851a57cd0c050c415
Building a tier 3 powerpc-unknown-linux-gnuspe <<-- testing code only in my local branch for now

option 1

Using default powerpc-unknown-linux-gnu along with RUSTFLAGS = -Ctarget-cpu=e500 lead to the exact same result, seftault at startup. I may not be using the right RUSTFLAGS?

option 2

As for option 2 I feel I'm digging my own hole as I'm unable to build a powerpc-unknown-linux-gnuspe target using Synology provided toolchain and toolkit. I'm able to build up to stage1 & stage2 but unable to create a fully working target using cargo along with either my stage1 or stage2 builds... (clearly there is something I'm not fully understanding). Here's what I have so far:

git clone --depth 1 https://github.com/rust-lang/rust.git
./x setup compiler
PATH="$(WORK_DIR)/$(TC_TARGET)/bin:$${PATH}" ./x build --target $(RUST_TARGET)
rustup toolchain link powerpc-stage1 $(WORK_DIR)/rust/build/host/stage1
PATH="$(WORK_DIR)/$(TC_TARGET)/bin:$${PATH}" ./x build --stage 2 --target $(RUST_TARGET))
rustup toolchain link pwoerpc-stage2 $(WORK_DIR)/rust/build/host/stage2
--->> Up to this point all working OK <<---

Where it then fails:

rustup override set nightly
echo "[llvm]" >> $(WORK_DIR)/rust/config.toml
echo "allow-old-toolchain = true" >> $(WORK_DIR)/rust/config.toml   <<-- Presuming as `powerpc-e500v2-linux-gnuspe-gcc` is v4.9.3
PATH="$(WORK_DIR)/$(TC_TARGET)/bin:$${PATH}" POWERPC_UNKNOWN_LINUX_GNUSPE_OPENSSL_DIR=$(WORK_DIR)/../../../toolkit/syno-$(ARCH)-$(TCVERSION)/work/usr/ RUST_BACKTRACE=full cargo +$(firstword $(subst -, ,$(RUST_TARGET)))-stage1 build -Zbuild-std=core,alloc --target powerpc-unknown-linux-gnuspe

The error relates to llvm (enven though it did built it succesfully during either stage1 or stage2):

error: failed to run custom build command for `rustc_llvm v0.0.0 (/home/spksrc/qoriq-debug/spksrc/toolchain/syno-qoriq-6.2.4/work/rust/compiler/rustc_llvm)`
Caused by:
  process didn't exit successfully: `/home/spksrc/qoriq-debug/spksrc/toolchain/syno-qoriq-6.2.4/work/rust/target/debug/build/rustc_llvm-dc7ac9c0f6cecf54/build-script-build` (exit status: 101)
  --- stdout
  cargo:rustc-check-cfg=values(llvm_component,"ipo")
  cargo:rustc-check-cfg=values(llvm_component,"bitreader")
  cargo:rustc-check-cfg=values(llvm_component,"bitwriter")
  cargo:rustc-check-cfg=values(llvm_component,"linker")
  cargo:rustc-check-cfg=values(llvm_component,"asmparser")
  cargo:rustc-check-cfg=values(llvm_component,"lto")
  cargo:rustc-check-cfg=values(llvm_component,"coverage")
  cargo:rustc-check-cfg=values(llvm_component,"instrumentation")
  cargo:rustc-check-cfg=values(llvm_component,"x86")
  cargo:rustc-check-cfg=values(llvm_component,"arm")
  cargo:rustc-check-cfg=values(llvm_component,"aarch64")
  cargo:rustc-check-cfg=values(llvm_component,"amdgpu")
  cargo:rustc-check-cfg=values(llvm_component,"avr")
  cargo:rustc-check-cfg=values(llvm_component,"loongarch")
  cargo:rustc-check-cfg=values(llvm_component,"m68k")
  cargo:rustc-check-cfg=values(llvm_component,"csky")
  cargo:rustc-check-cfg=values(llvm_component,"mips")
  cargo:rustc-check-cfg=values(llvm_component,"powerpc")
  cargo:rustc-check-cfg=values(llvm_component,"systemz")
  cargo:rustc-check-cfg=values(llvm_component,"jsbackend")
  cargo:rustc-check-cfg=values(llvm_component,"webassembly")
  cargo:rustc-check-cfg=values(llvm_component,"msp430")
  cargo:rustc-check-cfg=values(llvm_component,"sparc")
  cargo:rustc-check-cfg=values(llvm_component,"nvptx")
  cargo:rustc-check-cfg=values(llvm_component,"hexagon")
  cargo:rustc-check-cfg=values(llvm_component,"riscv")
  cargo:rustc-check-cfg=values(llvm_component,"bpf")
  cargo:rerun-if-env-changed=RUST_CHECK
  cargo:rerun-if-env-changed=REAL_LIBRARY_PATH_VAR

  --- stderr
  thread 'main' panicked at compiler/rustc_llvm/build.rs:51:59:
  REAL_LIBRARY_PATH_VAR
  stack backtrace:
     0: rust_begin_unwind
     1: core::panicking::panic_fmt
     2: core::option::expect_failed
     3: core::option::Option<T>::expect
               at /home/spksrc/qoriq-debug/spksrc/toolchain/syno-qoriq-6.2.4/work/rust/library/core/src/option.rs:888:21
     4: build_script_build::restore_library_path
               at ./build.rs:51:15
     5: build_script_build::main
               at ./build.rs:113:5
     6: core::ops::function::FnOnce::call_once
               at /home/spksrc/qoriq-debug/spksrc/toolchain/syno-qoriq-6.2.4/work/rust/library/core/src/ops/function.rs:250:5
  note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
warning: build failed, waiting for other jobs to finish...
make[2]: *** [../../mk/spksrc.tc-rust.mk:64: rustc_target] Error 101
/home/spksrc/qoriq-debug/spksrc/cross/bat/work-qoriq-6.2.4/tc_vars.mk:1: *** An error occured while setting up the toolchain, please check the messages above.  Stop.
make[1]: Leaving directory '/home/spksrc/qoriq-debug/spksrc/cross/bat'

option 3

Now, further reading and as option 3 I may be able to "rebuild" tier 1 powerpc-unknown-linux-gnu by adding the proper target features such as https://github.com/rust-lang/rust/issues/117347 . But guessing I'd have to run through the same procession as option 2 ?

saethlin commented 10 months ago

Using default powerpc-unknown-linux-gnu along with RUSTFLAGS = -Ctarget-cpu=e500 lead to the exact same result, seftault at startup. I may not be using the right RUSTFLAGS?

You are using a tier 2 target. Can you compile any programs successfully? It's not clear to me from your description that this is not a miscompile.

th0ma7 commented 10 months ago

You are using a tier 2 target. Can you compile any programs successfully? It's not clear to me from your description that this is not a miscompile.

@saethlin nope, no programs seems to runs on the qoriq platform using powerpc-unknown-linux-gnu with or without RUSTFLAGS = -Ctarget-cpu=e500. Which is why I'm wondering if I'm using the right target? Or missing build flags... ?

workingjubilee commented 10 months ago

Try -Ctarget-cpu=e500mc

workingjubilee commented 10 months ago

I'm confused, you said QoriQ, but then you said you're targeting MPC8548? Which is it? No, they're not necessarily guaranteed to be compatible: ignore the vendor's claims otherwise and approach this empirically.

workingjubilee commented 10 months ago

You use a vendor-provided (and thus possibly patched?) gcc, so it seems plausible to me that LLVM might have a different model of the CPU capabilities. Can you use clang to build, instead of gcc, for this target, and get working binaries? Or even a stock gcc?

You also made a typo: rustup toolchain link pwoerpc-stage2 $(WORK_DIR)/rust/build/host/stage2

th0ma7 commented 10 months ago

You use a vendor-provided (and thus possibly patched?) gcc, so it seems plausible to me that LLVM might have a different model of the CPU capabilities.

Indeed, Synology provide a GCC toolchain for each arch. The qoriq (e.g. powerpc) is using a version 4.9.3 of gcc.

Can you use clang to build, instead of gcc, for this target, and get working binaries? Or even a stock gcc?

That would require me to build a clang (or gcc) cross-compiler for this arch? Probably feasible but, I did try in the past using gcc in order to have a newer (and identical) version for all targets but I did not managed to assemble all the pieces together properly...

workingjubilee commented 10 months ago

LLVM-based compilers are always cross-compilers, that I've seen? You can use an x86_64-linux clang, or likewise for rustc, to emit code for PowerPC. The usual clang builds include codegen components for all their supported targets, I think.

workingjubilee commented 10 months ago

Indeed, Synology provide a GCC toolchain for each arch. The qoriq (e.g. powerpc) is using a version 4.9.3 of gcc.

Regarding this: do they patch it?

th0ma7 commented 10 months ago

Indeed, Synology provide a GCC toolchain for each arch. The qoriq (e.g. powerpc) is using a version 4.9.3 of gcc.

Regarding this: do they patch it?

No clue. And I was reviewing our own native llvm build that we provide part of synocli-devel package, we're dropping down to version 9 of llvm for archs that uses gcc <= 5.1... Which includes qoriq whereas it won't be able to build it anyhow without another version of GCC (or clang for that matter).

workingjubilee commented 10 months ago

Rust ships LLVM 17.

workingjubilee commented 10 months ago

GCC is a GPL project, so if you have been given the software, you also have the right to demand the source code, including its patches. ( also if you saw any messages between this and "Rust ships LLVM 17" from me: I'm blaming them all on the cat finding a few macro-commands and the enter key. )

th0ma7 commented 9 months ago

@workingjubilee so while my cycles where rather limited I did manage to make a bit of mileage over the last month...

In my current PR https://github.com/SynoCommunity/spksrc/pull/5879 I was able to automate building a rustc compiler supporting powerpc-unknown-linux-gnuspe.

Why you might ask? because it hapens that resulting binaries contains lwsync instead of sync (ref: https://github.com/rust-lang/rust/issues/96394#issuecomment-1113033290). It was confirmed below in the thread that using -Ctarget-cpu=e500 in conjunction with tier 3 powerpc-unknown-linux-gnuspe target support solves the issue for exactly the the Synology NAS model and CPU type I'm having issues with (ref: https://github.com/rust-lang/rust/issues/96394#issuecomment-1113051115).

So now in one hand I do have a rustc compiler with tier 3 powerpc-unknown-linux-gnuspe target support... And in the other I'm passing -Ctarget-cpu=e500 using RUSTFLAGS variable... and I can see it being used in the build output (and compiler doesn't complain).

But sadly the end-result binaries still contains lwsync (ref: https://github.com/SynoCommunity/spksrc/pull/5879#issuecomment-1837618214) and ends-up segfaulting (as to be expected, ref: https://github.com/SynoCommunity/spksrc/issues/5847#issuecomment-1837591884)

Question I have, what am I missing?

If someone else stumbles on this and wonder howto:

    @(cd $(WORK_DIR) && [ ! -d rust ] && git clone --depth 1 https://github.com/rust-lang/rust.git || true)
    @(cd $(WORK_DIR)/rust && ./x setup compiler)
    @(cd $(WORK_DIR)/rust && \
        CFLAGS_$(subst -,_,$(RUST_TARGET))="$(TC_EXTRA_CFLAGS)" \
        CXXFLAGS_$(subst -,_,$(RUST_TARGET))="$(TC_EXTRA_CFLAGS)" \
        LDFLAGS_$(subst -,_,$(RUST_TARGET))="--sysroot=$(WORK_DIR)/$(TC_TARGET)/$(TC_SYSROOT)" \
        RUST_BACKTRACE=full \
        ./x build --config $(TC_LOCAL_VARS_RUST))
    @rustup toolchain link $(TC_RUSTUP_TOOLCHAIN) $(WORK_DIR)/rust/build/host/stage$(RUSTUP_DEFAULT_TOOLCHAIN_STAGE)

Where my config file refers to Synology toolchain tools:

profile = "compiler"

[build]
target = ["powerpc-unknown-linux-gnuspe"]
build-stage = 2
docs = false
docs-minification = false
compiler-docs = false

[target.powerpc-unknown-linux-gnuspe]
cc = "/home/spksrc/qoriq-debug/spksrc/toolchain/syno-qoriq-6.2.4/work/powerpc-e500v2-linux-gnuspe/bin/powerpc-e500v2-l
inux-gnuspe-gcc"
cxx = "/home/spksrc/qoriq-debug/spksrc/toolchain/syno-qoriq-6.2.4/work/powerpc-e500v2-linux-gnuspe/bin/powerpc-e500v2-
linux-gnuspe-g++"
ar = "/home/spksrc/qoriq-debug/spksrc/toolchain/syno-qoriq-6.2.4/work/powerpc-e500v2-linux-gnuspe/bin/powerpc-e500v2-l
inux-gnuspe-ar"
ranlib = "/home/spksrc/qoriq-debug/spksrc/toolchain/syno-qoriq-6.2.4/work/powerpc-e500v2-linux-gnuspe/bin/powerpc-e500
v2-linux-gnuspe-ranlib"
linker = "/home/spksrc/qoriq-debug/spksrc/toolchain/syno-qoriq-6.2.4/work/powerpc-e500v2-linux-gnuspe/bin/powerpc-e500
v2-linux-gnuspe-gcc"

th0ma7 commented 8 months ago

@workingjubilee and @saethlin and @wycats I believe I have found where the issue is ... When building my powerpc-unknown-linux-gnuspe rust toolchain support I am using Synology provided toolchain and thus pointing to its relevant cc and all as per the config.toml above.

The rust toolchain do build just fine, but I noticed that lwsync do appear in the resulting libraries rust/build/host/stage1/lib/rustlib/powerpc-unknown-linux-gnuspe/lib/libstd-4b4243768d8e8241.so whereas objdump -d gives a few of those which are incompatible with this platform:

   ac98c:   7c 20 04 ac     lwsync
   ac9a8:   7c 20 04 ac     lwsync

My theory is that when building any software afterward and linking to these libraries, it will necessarely fail as result holds incompatible lwsync ...

Now why? I am providing the necessary CFLAGS and CXXFLAGS at rust toolchain build time... so issue must be how the flags are being passed in here somewhere where flags are not being honored:

(cd /home/spksrc/qoriq-debug-update/spksrc/toolchain/syno-qoriq-6.2.4/work/rust \
   CFLAGS_powerpc_unknown_linux_gnuspe="-mcpu=8548 -mhard-float -mfloat-gprs=double" \
   CXXFLAGS_powerpc_unknown_linux_gnuspe="-mcpu=8548 -mhard-float -mfloat-gprs=double" \
   LDFLAGS_powerpc_unknown_linux_gnuspe="--sysroot=/home/spksrc/qoriq-debug-update/spksrc/toolchain/syno-qoriq-6.2.4/work/powerpc-e500v2-linux-gnuspe/powerpc-e500v2-linux-gnuspe/sysroot" \
   RUST_BACKTRACE=full \
   ./x build --config /home/spksrc/qoriq-debug-update/spksrc/toolchain/syno-qoriq-6.2.4/work/qoriq.toml)

Help would be much appreciated, thnx in advance!

glaubitz commented 8 months ago

Support for powerpc-unknown-linux-gnuspe was added to Rust by me and always worked fine until the port was removed from Debian due to gcc upstream dropping support for this target.

th0ma7 commented 7 months ago

@glaubitz perhaps you can assist in providing me hints as to why things are not working out?

quick background, synocommunity is spksrc framework github project to build package for Synology NAS on multiple archs (armv5, v7, v8, x86_64, ppc, etc). All other archs are well supported from a rust point of view with the exception of qoriq being a powerpc gnuspe tier 3 target. My current PR in order to support rust for qoriq is https://github.com/SynoCommunity/spksrc/pull/5879

To build rust support I'm using Synology DSM 6.2.4 toolchain for that specific arch. Toolchain bundle file is located here https://sourceforge.net/projects/dsgpl/files/Tool%20Chain/DSM%206.2.4%20Tool%20Chains/PowerPC%20QorIQ%20Linux%202.6.32/qoriq-gcc493_glibc220_hard_qoriq-GPL.txz/download

gcc provided part of the toolchain is gcc version 4.9.3 20150311 (prerelease) (crosstool-NG 1.20.0)

Configuration file I'm using is the following:

profile = "compiler"

[build]
target = ["x86_64-unknown-linux-gnu", "powerpc-unknown-linux-gnuspe"]
build-stage = 1
doc-stage = 2
docs = false
docs-minification = false
compiler-docs = false

[rust]
channel = "stable"
lto = "off"

[llvm]
download-ci-llvm = "if-unchanged"

[install]

[dist]

[target.x86_64-unknown-linux-gnu]

[target.powerpc-unknown-linux-gnuspe]
cc = "/home/spksrc/qoriq-debug-update/spksrc/toolchain/syno-qoriq-6.2.4/work/powerpc-e500v2-linux-gnuspe/bin/powerpc-e500v2-linux-gnuspe-gcc"
cxx = "/home/spksrc/qoriq-debug-update/spksrc/toolchain/syno-qoriq-6.2.4/work/powerpc-e500v2-linux-gnuspe/bin/powerpc-e500v2-linux-gnuspe-g++"
ar = "/home/spksrc/qoriq-debug-update/spksrc/toolchain/syno-qoriq-6.2.4/work/powerpc-e500v2-linux-gnuspe/bin/powerpc-e500v2-linux-gnuspe-ar"
ranlib = "/home/spksrc/qoriq-debug-update/spksrc/toolchain/syno-qoriq-6.2.4/work/powerpc-e500v2-linux-gnuspe/bin/powerpc-e500v2-linux-gnuspe-ranlib"
linker = "/home/spksrc/qoriq-debug-update/spksrc/toolchain/syno-qoriq-6.2.4/work/powerpc-e500v2-linux-gnuspe/bin/powerpc-e500v2-linux-gnuspe-gcc"

Arch gcc specific flags we've been using since ever are -mcpu=8548 -mhard-float -mfloat-gprs=double. Working equivalent to be used with RUSTFLAGS for the llvm backend is -Ctarget-cpu=e500 based on https://github.com/rust-lang/rust/issues/96394#issuecomment-1113271471

Hopefully what I'm missing is obvious... but your help would be more than welcomed ...

rust-lang / rust