Open thomas725 opened 9 months ago
commenting out the panic=
property in this Cargo.toml
block
[profile.release]
opt-level = 'z' # Optimize for size
lto = true # Enable link-time optimization
codegen-units = 1 # Reduce number of codegen units to increase optimizations
#panic = 'abort' # Abort on panic
strip = true # Strip symbols from binary*
makes this release build command work too:
CROSS_CONTAINER_ENGINE=podman RUSTFLAGS="-C target-feature=-crt-static" cross +nightly build --target powerpc-unknown-linux-musl --release
and results in a 321kb big binary:
ELF 32-bit MSB pie executable, PowerPC or cisco 4500, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-powerpc.so.1, stripped
Running that binary on my openwrt router sadly doesn't print "hello world" as expected, but instead:
-ash: /bin/my-project: not found
I guess that's because it's looking for the first while we only have the second on device:
root@openwrt:~# ls -alih /lib/ld-musl-powerpc.so.1
ls: /lib/ld-musl-powerpc.so.1: No such file or directory
root@openwrt:~# ls -alih /lib/ld-musl-powerpc-sf.so.1
305 lrwxrwxrwx 1 root root 7 Oct 15 2022 /lib/ld-musl-powerpc-sf.so.1 -> libc.so
ohhh, but it actually is compatible! running
ln -s /lib/libc.so /lib/ld-musl-powerpc.so.1
makes my binary print the expected output!
root@openwrt:~# /bin/my-project
hello world
@Emilgardis Can you help me translate our success into something I can put into a .gitlab-ci.yml
task?
I'm reading https://github.com/cross-rs/cross/issues/273 but am a bit confused in how complex that task seems to be. Do I really need to run docker in docker, can't I just use the build image directly?
also, will this solution blow up as soon as I do anything that uses float-point operations? since my musl-libc is actually soft-float but the compiler seems to think we have hardware-float?
Can you make the second = soft-float image public so I can use that one instead? Also can you merge what we managed to do into the regular cross-rs infrastructure?
Awesome!
Cross works by running in/starting a container, so unfortunately container in container is needed. If you want you can skip cross and just emulate what cross does but without cross. Shouldn't be too hard, just copy the dockerfile I've made and supply all the needed files and use that as the base image for the runner, then also install rustup and you should be all good :)
If you do want to use cross for Gitlab Runners, recommended usage is via CROSS_REMOTE
now. see https://github.com/cross-rs/cross/wiki/FAQ#gitlab-ci
also, could you try adding a float operation to the binary, interested if the device actually can do hard floats.
something like
println!("{:?}", (1.0f32/10.0).sin());
no, doesn't seem like it can:
hello world Illegal instruction
running in dev environment this instead prints:
hello world 0.09983342
I will have a look at https://github.com/cross-rs/cross/wiki/FAQ#gitlab-ci tomorrow and see if I can make that work, thanks for your great support, you're amazing :) gotta get some sleep now.
Okey, so I don't think what's described at https://github.com/cross-rs/cross/wiki/FAQ#gitlab-ci can work for me, since I'm not hosting a gitlab instance myself but just using the public gitlab.com instance which includes 400 free CI minutes myself - but I don't think (or know how) I can access a docker host from there like described at tcp://docker:2375/
I'm considering using github instead, but the section about github says it need's to be reworked because actions-rs is unmaintained.
You should be able to use tcp://docker:2375/
just fine, doesnt matter if you host the instance yourself or not.
So, for only using softfloat, you can do RUSTFLAGS="-C target-feature=-crt-static,-hard-float"
and also use the muslsf
image instead.
Another thing you could do is use a custom target, this way you don't have to specify any rustflags via env.
#Cross.toml
[target."powerpc-unknown-linux-muslsf.json"]
image = "ghcr.io/emilgardis/powerpc-unknown-linux-muslsf-cross:test"
build-std = true
# powerpc-unknown-linux-muslsf.json
{
"arch": "powerpc",
"crt-objects-fallback": "musl",
"crt-static-default": false,
"crt-static-respected": true,
"data-layout": "E-m:e-p:32:32-Fn32-i64:64-n32",
"dynamic-linking": true,
"env": "musl",
"has-rpath": true,
"has-thread-local": true,
"linker-flavor": "gnu-cc",
"llvm-target": "powerpc-unknown-linux-musl",
"max-atomic-width": 32,
"os": "linux",
"position-independent-executables": true,
"pre-link-args": {
"gnu-cc": [
"-m32"
],
"gnu-lld-cc": [
"-m32"
]
},
"relro-level": "full",
"stack-probes": {
"kind": "inline"
},
"supported-split-debuginfo": [
"packed",
"unpacked",
"off"
],
"target-endian": "big",
"target-family": [
"unix"
],
"target-mcount": "_mcount",
"target-pointer-width": "32",
"features": "-hard-float"
}
cross +nightly build --release --target powerpc-unknown-linux-muslsf.json
looking further into the MPC8540, you might be fine with SPE. this comment is a bit concerning though
The following CPUs are known to not working with this port:
MPC8540 (for this one, you can use Debian's regular powerpc port with activated FPU emulation in the kernel)
#Cross.toml
[target."powerpc-unknown-linux-muslspe.json"]
image = "ghcr.io/emilgardis/powerpc-unknown-linux-muslsf-cross:test"
build-std = true
# powerpc-unknown-linux-muslspe.json
{
"arch": "powerpc",
"crt-objects-fallback": "musl",
"crt-static-default": false,
"crt-static-respected": true,
"data-layout": "E-m:e-p:32:32-i64:64-n32",
"dynamic-linking": true,
"env": "musl",
"has-rpath": true,
"has-thread-local": true,
"linker-flavor": "gnu-cc",
"llvm-target": "powerpc-unknown-linux-muslspe",
"max-atomic-width": 32,
"os": "linux",
"position-independent-executables": true,
"pre-link-args": {
"gnu-cc": [
"-m32 -mspe"
],
"gnu-lld-cc": [
"-m32 -mspe"
]
},
"relro-level": "full",
"stack-probes": {
"kind": "inline"
},
"supported-split-debuginfo": [
"packed",
"unpacked",
"off"
],
"target-endian": "big",
"target-family": [
"unix"
],
"target-mcount": "_mcount",
"target-pointer-width": "32",
"features": "-fpu,+spe"
}
hmm, something with that custom target json didn't work as hoped:
$ cross +nightly build --target powerpc-unknown-linux-muslsf.json --release
[cross] warning: using newer rustc `1.75.0-nightly (2f1bd0729 2023-10-27)` for the target. Current active rustc on the host is `rustc 1.73.0 (cc66ad468 2023-10-03)`.
> Update with `rustup update`
Compiling my-project v0.1.0 (/home/user/Development/git/my-project)
error: linking with `cc` failed: exit status: 1
|
= note: LC_ALL="C" PATH="/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin" VSLANG="1033" "cc" "-m32" "/tmp/rustctgSOPN/symbols.o" "/target/powerpc-unknown-linux-muslsf/release/deps/my_project-b43545ee96f7e2e3.my_project.2be8c3d152745613-cgu.0.rcgu.o" "-Wl,--as-needed" "-L" "/target/powerpc-unknown-linux-muslsf/release/deps" "-L" "/target/release/deps" "-L" "/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/powerpc-unknown-linux-muslsf/lib" "-Wl,-Bstatic" "/target/powerpc-unknown-linux-muslsf/release/deps/libcompiler_builtins-ef09095d5d60da69.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/powerpc-unknown-linux-muslsf/lib" "-o" "/target/powerpc-unknown-linux-muslsf/release/deps/my_project-b43545ee96f7e2e3" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,--strip-all" "-nodefaultlibs"
= note: /usr/bin/ld: cannot find Scrt1.o: No such file or directory
/usr/bin/ld: cannot find crti.o: No such file or directory
...
/usr/bin/ld: /target/powerpc-unknown-linux-muslsf/release/deps/my_project-b43545ee96f7e2e3.my_project.2be8c3d152745613-cgu.0.rcgu.o: relocations in generic ELF (EM: 20)
/usr/bin/ld: /target/powerpc-unknown-linux-muslsf/release/deps/my_project-b43545ee96f7e2e3.my_project.2be8c3d152745613-cgu.0.rcgu.o: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status
and the spe version fails like this:
error: linking with `cc` failed: exit status: 1
|
= note: LC_ALL="C" PATH="/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin" VSLANG="1033" "cc" "-m32 -mspe" "/tmp/rustcGLJTbP/symbols.o" "/target/powerpc-unknown-linux-muslspe/release/deps/my_project-5fc72a3857ce6d3c.my_project.d68e5b2406453db7-cgu.0.rcgu.o" "-Wl,--as-needed" "-L" "/target/powerpc-unknown-linux-muslspe/release/deps" "-L" "/target/release/deps" "-L" "/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/powerpc-unknown-linux-muslspe/lib" "-Wl,-Bstatic" "/target/powerpc-unknown-linux-muslspe/release/deps/libcompiler_builtins-5290246e7a0b36d3.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/powerpc-unknown-linux-muslspe/lib" "-o" "/target/powerpc-unknown-linux-muslspe/release/deps/my_project-5fc72a3857ce6d3c" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,--strip-all" "-nodefaultlibs"
= note: cc: error: unrecognized command line option '-m32 -mspe'
okey so the binary resulting from this command:
RUSTFLAGS="-C target-feature=-crt-static,-hard-float" CROSS_CONTAINER_ENGINE=podman cross +nightly build --target powerpc-unknown-linux-musl --release
gives good float output:
hello world 0.09983342
even though it does output this warning during building:
warning: unknown feature specified for `-Ctarget-feature`: `hard-float`
|
= note: it is still passed through to the codegen backend
= help: consider filing a feature request
that warning is what I'm discussing in https://github.com/rust-lang/rust/issues/117347
for the powerpc-unknown-linux-muslsf.json
and powerpc-unknown-linux-muslspe.json
targets, forgot that we also need to specify the linker
i'm not certain what the envvar would be called but you can set the linker in .cargo/config.toml
should be powerpc-linux-muslsf-gcc
I've tried putting those two into .cargo/config.toml
:
[target."powerpc-unknown-linux-muslsf.json"]
linker = "powerpc-linux-muslsf-gcc"
[target.powerpc-unknown-linux-musl]
linker = "powerpc-linux-muslsf-gcc"
But it didn't change the output of cross +nightly build --target powerpc-unknown-linux-muslsf.json --release
shown 4 posts above.
btw. huge thanks for helping me with this! :)
Also, since you said using tcp://docker:2375/
on public gitlab.com should work just fine, I wanted to try that - but I need a docker image for the layer calling cross, and the guide at https://github.com/cross-rs/cross/wiki/FAQ#gitlab-ci doesn't specify one. Is there one provided by the cross-rs project for this purpose or do I need to do cargo install cross
in a generic rust docker image?
we have a generic image https://github.com/cross-rs/cross/pkgs/container/cross
thanks! is there also one with rust nightly pre-installed?
$ RUSTFLAGS="-C target-feature=-crt-static,-hard-float" cross +nightly build --target powerpc-unknown-linux-musl --release
[cross] warning: unable to get metadata for package
[cross] note: Falling back to `cargo` on the host.
error: toolchain 'nightly-x86_64-unknown-linux-gnu' is not installed
no wouldn't make sense to install nightly in the images since they are "nightly"
install nightly with rustup install nightly
also, can you answer my question in https://github.com/rust-lang/rust/issues/117347#issuecomment-1784074780 please :3
Okey, I did, but now get this strange error:
$ RUSTFLAGS="-C target-feature=-crt-static,-hard-float" cross +nightly build --target powerpc-unknown-linux-musl --release
[cross] warning: using newer rustc `1.75.0-nightly (608e9682f 2023-10-29)` for the target. Current active rustc on the host is `rustc 1.73.0 (cc66ad468 2023-10-03)`.
> Update with `rustup update`
info: downloading component 'rust-src'
info: installing component 'rust-src'
Error:
0: `docker inspect runner-xs6vzpvo-project-51146970-concurrent-0` failed with exit status: 1
Stderr:
Error: No such object: runner-xs6vzpvo-project-51146970-concurrent-0
that's https://github.com/cross-rs/cross/issues/1351, im still not sure why it happens since it has worked for a while before
okey, thanks for showing me the connection! So cross-rs build won't work on GitLab until there's a solution for #1351 - correct?
Can you help me get the json based build definition files working?
For the powerpc-unknown-linux-muslsf.json
you posted I get this error:
/usr/bin/ld: cannot find Scrt1.o: No such file or directory /usr/bin/ld: cannot find crti.o: No such file or directory
And for the *spe.json
version's the error is:
cc: error: unrecognized command line option '-m32 -mspe'
the powerpc-unknown-linux-muslsf.json
target is using the wrong c toolchain, you'd have to tweak the variables set in
however, I'm not sure about the naming of the env-vars with .json targets, I think I remember something about cargo not supporting them
for the spe version, same story
For the last few weeks with the help of @Emilgardis I've tried to get rust compiled for my powerpc openwrt device.
TP-Link TL-WDR4900 v1.x
See also: https://github.com/cross-rs/cross/pull/1349
Currently we're testing those images created by @Emilgardis:
We managed to get the first one of those to produce debug build using:
but since the resulting binary is 15mb big I'd need a release build to actually test if it can run on my device, which failed to build so far: