Validate that all combinations of BUILDPLATFORM, TARGETPLATFORM, Dockerfile, and host CPU architecture produce correct executable

rucoder commented 3 months ago

With some combinations of BUILDPLATFORM, TARGETPLATFORM, Dockerfile, and host CPU architecture downstream rust applications are built for incorrect CPU architecture.

[x] Identify broken combinations fixed by https://github.com/lf-edge/eve-rust/pull/10
[ ] Add tests to validate it https://github.com/lf-edge/eve-rust/pull/9
[ ] run tests on CI

deitch commented 3 months ago

@rucoder do you have an example to recreate it? I think @jsfakian also succeeded in creating the issue? I am not asking about the fix, just to recreate the problem.

rucoder commented 3 months ago

this is the example Ioannis sent me yesterday, but I have not tried it yet

# syntax=docker/dockerfile-upstream:1.5.0-rc2-labs
#FROM --platform=$BUILDPLATFORM jsfakian/eve-rust:1.80.1 AS rust-host
#ARG TARGETARCH
FROM jsfakian/eve-rust:1.80.1 AS rust

# map Docker's $TARGETARCH to Rust target
#FROM rust-host AS target-amd64
#ENV CARGO_BUILD_TARGET="x86_64-unknown-linux-musl"

#FROM rust-host AS target-arm64
#ENV CARGO_BUILD_TARGET="aarch64-unknown-linux-musl"

#FROM rust-host AS target-riscv64
#ENV CARGO_BUILD_TARGET="riscv64gc-unknown-linux-gnu"

#FROM target-$TARGETARCH AS rust

WORKDIR /usr/local/my-installer
COPY . .

# Add the vendor tar file
RUN tar xvf /usr/local/my-installer/vendor.tar.gz \
    && rm vendor.tar.gz

# tools already in place
RUN cargo build --release --verbose
RUN cargo sbom > sbom.spdx.json

RUN cp /usr/local/my-installer/target/$CARGO_BUILD_TARGET/release/installer /usr/local/my-installer/target/installer

FROM lfedge/eve-alpine:1f7685f95a475c6bbe682f0b976f12180b6c8726 AS build
SHELL ["/bin/ash", "-eo", "pipefail", "-c"]
ENV PKGS ncurses-dev smartmontools mtools strace util-linux e2fsprogs e2fsprogs-extra jq

RUN eve-alpine-deploy.sh

COPY --from=rust /usr/local/my-installer/target/installer /out/sbin/installer
COPY --from=rust /usr/local/my-installer/sbom.spdx.json /out/sbin/sbom.spdx.json

FROM scratch
COPY --from=build /out/ /
ENV RUST_BACKTRACE full
COPY run-installer.sh /sbin/run-installer.sh

WORKDIR /
ENTRYPOINT ["/sbin/run-installer.sh"]

and the error message


2.177 error: linking with `/usr/bin/clang` failed: exit status: 1
2.177   |
2.177   = note: LC_ALL="C" PATH="/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin:/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin:/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/bin:/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" VSLANG="1033" "/usr/bin/clang" "-Wl,--version-script=/tmp/rustc4y4cU5/list" "-Wl,--no-undefined-version" "-m64" "/tmp/rustc4y4cU5/symbols.o" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.rustversion.48f492ba9fdb42b3-cgu.0.rcgu.o" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.rustversion.48f492ba9fdb42b3-cgu.1.rcgu.o" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.rustversion.48f492ba9fdb42b3-cgu.2.rcgu.o" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.rustversion.48f492ba9fdb42b3-cgu.3.rcgu.o" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.rustversion.48f492ba9fdb42b3-cgu.4.rcgu.o" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.rustversion.48f492ba9fdb42b3-cgu.5.rcgu.o" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.rustversion.48f492ba9fdb42b3-cgu.6.rcgu.o" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.rustversion.48f492ba9fdb42b3-cgu.7.rcgu.o" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.1kzo4sn02q0yb1bqsb0prf8ua.rcgu.rmeta" "/usr/local/my-installer/target/release/deps/rustversion-6b2bbacd6ab242a4.72xrpt734hw9o2axvytp0dovp.rcgu.o" "-Wl,--as-needed" "-L" "/usr/local/my-installer/target/release/deps" "-L" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libproc_macro-bb90118a9d3a1120.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-11278e6f292a83a2.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libpanic_unwind-1f0a0dc4faeec0a3.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libobject-aa008ac291cf6c9f.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libmemchr-222e4867c5d474c8.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libaddr2line-b729ba7443944882.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libgimli-59743343eaf025ae.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_demangle-cb5149dfbee56ccb.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd_detect-e57a974822694b25.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libhashbrown-a2863812bbdf5e6e.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_alloc-925e49df7b4ee2bb.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libminiz_oxide-fb6cd229e1eac996.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libadler-22b25a5ddfc02a07.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libunwind-bf17bea8f84cb8cb.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcfg_if-218ac2e8053d7769.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-c8a3bb72f2547d7e.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/liballoc-a6f924a0e16819b9.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/librustc_std_workspace_core-aedcb87a10c080a3.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcore-c1cee64659224c4c.rlib" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-b35caaccb3ca2afc.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib" "-o" "/usr/local/my-installer/target/release/deps/librustversion-6b2bbacd6ab242a4.so" "-Wl,--gc-sections" "-shared" "-Wl,-z,relro,-z,now" "-Wl,--strip-all" "-nodefaultlibs" "--ld-path=/usr/bin/mold" "--target=x86_64-unknown-linux-musl"
2.177   = note: mold: fatal: cannot open crtbeginS.o: No such file or directory
2.177           clang: error: linker command failed with exit code 1 (use -v to see invocation)
2.177
2.177
2.195    Compiling crossbeam-channel v0.5.13
2.195      Running `/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/bin/rustc --crate-name crossbeam_channel --edition=2021 /usr/local/my-installer/vendor/crossbeam-channel/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=z -C panic=abort -C linker-plugin-lto -C codegen-units=1 --warn=unexpected_cfgs --cfg 'feature="default"' --cfg 'feature="std"' --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values("default", "std"))' -C metadata=3e219331d488bf74 -C extra-filename=-3e219331d488bf74 --out-dir /usr/local/my-installer/target/release/deps -C linker=/usr/bin/clang -C strip=symbols -L dependency=/usr/local/my-installer/target/release/deps --extern crossbeam_utils=/usr/local/my-installer/target/release/deps/libcrossbeam_utils-205a64670172d02c.rmeta --cap-lints allow -C link-arg=--ld-path=/usr/bin/mold -C link-arg=--target=x86_64-unknown-linux-musl`
2.222 error: could not compile `rustversion` (lib) due to 1 previous error
2.222
2.222 Caused by:
2.222   process didn't exit successfully: `/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/bin/rustc --crate-name rustversion --edition=2018 /usr/local/my-installer/vendor/rustversion/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debug-assertions=off --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=6b2bbacd6ab242a4 -C extra-filename=-6b2bbacd6ab242a4 --out-dir /usr/local/my-installer/target/release/deps -C linker=/usr/bin/clang -C strip=symbols -L dependency=/usr/local/my-installer/target/release/deps --extern proc_macro --cap-lints allow -C link-arg=--ld-path=/usr/bin/mold -C link-arg=--target=x86_64-unknown-linux-musl --check-cfg 'cfg(cfg_macro_not_allowed)' --check-cfg 'cfg(host_os, values("windows"))'` (exit status: 1)
2.222 warning: build failed, waiting for other jobs to finish...

deitch commented 3 months ago

What platform was he running it on? How did he call it? And in what scenario does it work?

rucoder commented 3 months ago

What platform was he running it on? How did he call it? And in what scenario does it work?

both x86 linux and Mac. @jsfakian , I think I know why it failed for you. Did you build a multiplatfrom image for eve-rust and push it to your dockerhub or you built only a single platfrom?

jsfakian commented 3 months ago

I was running on x86 Linux and Mac. I built a multiplatform image for eve-rust and pushed it to my docker hub.

deitch commented 3 months ago

@jsfakian and I did a test. If you do a simple build, it fails. If you set the env var (like the Dockerfile), it fails. It fails on building proc-macro-error-attr. Here are the success and failure lines:

Success with env var export CARGO_BUILD_TARGET="x86_64-unknown-linux-musl":

/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/bin/rustc --crate-name proc_macro_error_attr --edition=2018 /usr/local/my-installer/vendor/proc-macro-error-attr/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=200 --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debug-assertions=off --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=48656eceb2ec36da -C extra-filename=-48656eceb2ec36da --out-dir /usr/local/my-installer/target/release/deps -C linker=/usr/bin/clang -C strip=symbols -L dependency=/usr/local/my-installer/target/release/deps --extern proc_macro2=/usr/local/my-installer/target/release/deps/libproc_macro2-f6ff1dbe72cec79b.rlib --extern quote=/usr/local/my-installer/target/release/deps/libquote-4ff18852125cfc7a.rlib --extern proc_macro --cap-lints allow

Failure without env var:

/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/bin/rustc --crate-name proc_macro_error_attr --edition=2018 /usr/local/my-installer/vendor/proc-macro-error-attr/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debug-assertions=off --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=48656eceb2ec36da -C extra-filename=-48656eceb2ec36da --out-dir /usr/local/my-installer/target/release/deps -C linker=/usr/bin/clang -C strip=symbols -L dependency=/usr/local/my-installer/target/release/deps --extern proc_macro2=/usr/local/my-installer/target/release/deps/libproc_macro2-f6ff1dbe72cec79b.rlib --extern quote=/usr/local/my-installer/target/release/deps/libquote-4ff18852125cfc7a.rlib --extern proc_macro --cap-lints allow -C link-arg=--ld-path=/usr/bin/mold -C link-arg=--target=x86_64-unknown-linux-musl

The only differences between the lines are:

success has: --diagnostic-width=200 (we do not think that matters)
failure has at the very end: -C link-arg=--ld-path=/usr/bin/mold -C link-arg=--target=x86_64-unknown-linux-musl

Same success if we do cargo build --target x86_64-unknown-linux-musl

We do not understand why explicitly setting CARGO_BUILD_TARGET causes that to happen. Since rust is on x86_64-unknown-linux-musl, wouldn't that be the default build target?

@rucoder thoughts?

deitch commented 3 months ago

More information. On failure, the linker says it is unable to find crtbeginS.o. This does exist on the platform, specifically at:

/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/<triple>/lib/self-contained/crtbeginS.o

E.g.:

/usr/local/my-installer # ls -l /usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/
total 6084
-rw-rw-rw-    1 root     root          3688 Aug  8 20:02 Scrt1.o
-rw-rw-rw-    1 root     root          3688 Aug  8 20:02 crt1.o
-rw-rw-rw-    1 root     root          3624 Aug  8 20:02 crtbegin.o
-rw-rw-rw-    1 root     root          3624 Aug  8 20:02 crtbeginS.o
-rw-rw-rw-    1 root     root          1056 Aug  8 20:02 crtend.o
-rw-rw-rw-    1 root     root          1056 Aug  8 20:02 crtendS.o
-rw-rw-rw-    1 root     root          2656 Aug  8 20:02 crti.o
-rw-rw-rw-    1 root     root          2600 Aug  8 20:02 crtn.o
-rw-rw-rw-    1 root     root       5982382 Aug  8 20:02 libc.a
-rw-rw-rw-    1 root     root        204164 Aug  8 20:02 libunwind.a
-rw-rw-rw-    1 root     root          4864 Aug  8 20:02 rcrt1.o

It looks like the linker is not finding these libraries to link, except when CARGO_BUILD_TARGET is set (or CLI flag). We wonder if maybe there is a big somewhere where it sets the lib path using the target but misses it if not explicit?

rucoder commented 3 months ago

@jsfakian and I did a test. If you do a simple build, it fails. If you set the env var (like the Dockerfile), it fails. It fails on building proc-macro-error-attr. Here are the success and failure lines:

Success with env var export CARGO_BUILD_TARGET="x86_64-unknown-linux-musl":

/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/bin/rustc --crate-name proc_macro_error_attr --edition=2018 /usr/local/my-installer/vendor/proc-macro-error-attr/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=200 --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debug-assertions=off --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=48656eceb2ec36da -C extra-filename=-48656eceb2ec36da --out-dir /usr/local/my-installer/target/release/deps -C linker=/usr/bin/clang -C strip=symbols -L dependency=/usr/local/my-installer/target/release/deps --extern proc_macro2=/usr/local/my-installer/target/release/deps/libproc_macro2-f6ff1dbe72cec79b.rlib --extern quote=/usr/local/my-installer/target/release/deps/libquote-4ff18852125cfc7a.rlib --extern proc_macro --cap-lints allow

Failure without env var:

/usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/bin/rustc --crate-name proc_macro_error_attr --edition=2018 /usr/local/my-installer/vendor/proc-macro-error-attr/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debug-assertions=off --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=48656eceb2ec36da -C extra-filename=-48656eceb2ec36da --out-dir /usr/local/my-installer/target/release/deps -C linker=/usr/bin/clang -C strip=symbols -L dependency=/usr/local/my-installer/target/release/deps --extern proc_macro2=/usr/local/my-installer/target/release/deps/libproc_macro2-f6ff1dbe72cec79b.rlib --extern quote=/usr/local/my-installer/target/release/deps/libquote-4ff18852125cfc7a.rlib --extern proc_macro --cap-lints allow -C link-arg=--ld-path=/usr/bin/mold -C link-arg=--target=x86_64-unknown-linux-musl

The only differences between the lines are:
* success has: `--diagnostic-width=200` (we do not think that matters)

* failure has at the very end: `-C link-arg=--ld-path=/usr/bin/mold -C link-arg=--target=x86_64-unknown-linux-musl`
Same success if we do cargo build --target x86_64-unknown-linux-musl

We do not understand why explicitly setting CARGO_BUILD_TARGET causes that to happen. Since rust is on x86_64-unknown-linux-musl, wouldn't that be the default build target?

@rucoder thoughts?

@deitch I think CARGO_BUILD_TARGET IS set but it is not -unknow-linux-musl because how would cargo know you want to build for musl? maybe try setting default toolchain via rustup?

e.g.

FROM tools-host AS target-arm64
RUN rustup default aarch64-unknown-linux-musl
ENV CARGO_BUILD_TARGET="aarch64-unknown-linux-musl"

UPDARE: no, toolchain has nothing todo with target. we MUST specify --target or CARGO_BUILD_TARGET to override the default target which matches the BUILD platform

deitch commented 3 months ago

@rucoder can you explain that?

The command that fails is:

 /usr/local/rustup/toolchains/1.80.1-x86_64-unknown-linux-musl/bin/rustc --crate-name proc_macro_error_attr --edition=2018 /installer/vendor/proc-macro-error-attr/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debug-assertions=off --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=48656eceb2ec36da -C extra-filename=-48656eceb2ec36da --out-dir /installer/target/release/deps -C linker=/usr/bin/clang -C strip=symbols -L dependency=/installer/target/release/deps --extern proc_macro2=/installer/target/release/deps/libproc_macro2-f6ff1dbe72cec79b.rlib --extern quote=/installer/target/release/deps/libquote-4ff18852125cfc7a.rlib --extern proc_macro --cap-lints allow -C link-arg=--ld-path=/usr/bin/mold -C link-arg=--target=x86_64-unknown-linux-musl

That looks like it is building for the local target. But who knows?

I think it is fine if our default target is local platform, and we set it. So maybe the dockerfile to build should include that cross-stuff, so it always sets the default target, and you can override it? We already do it for building the cargo tools, maybe we should leave the ENV set? That would fix it.

deitch commented 3 months ago

@rucoder I think we can close this now?

rucoder commented 3 months ago

@rucoder I think we can close this now?

So we are not going to add tests/exaples ? 😉

deitch commented 3 months ago

Ok, fine.

lf-edge / eve-rust

Validate that all combinations of BUILDPLATFORM, TARGETPLATFORM, Dockerfile, and host CPU architecture produce correct executable #8