search

rust-lang / cargo

The Rust package manager
https://doc.rust-lang.org/cargo
Apache License 2.0
12.64k stars 2.4k forks source link

Build fails if target is passed within`RUSTFLAGS` #11452

Open jpalus opened 1 year ago

jpalus commented 1 year ago

Problem

Trying to build delta 0.15.1 with:

CC=gcc RUSTFLAGS="--target thumbv7neon-unknown-linux-gnueabihf" cargo build --verbose -j 1

fails with:

   Compiling serde_derive v1.0.138
     Running `rustc --crate-name build_script_build /home/users/builder/.cargo/registry/src/github.com-1285ae84e5963aae/serde_derive-1.0.138/build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="default"' -C metadata=819ff560c5407cd8 -C extra-filename=-819ff560c5407cd8 --out-dir /home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/build/serde_derive-819ff560c5407cd8 -L dependency=/home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps --cap-lints allow --target thumbv7neon-unknown-linux-gnueabihf`
     Running `/home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/build/serde_derive-819ff560c5407cd8/build-script-build`
     Running `/home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/build/memchr-9e1e3f4935ac1ecb/build-script-build`
   Compiling jobserver v0.1.21
     Running `rustc --crate-name jobserver --edition=2018 /home/users/builder/.cargo/registry/src/github.com-1285ae84e5963aae/jobserver-0.1.21/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 -C metadata=841f90be8fa84132 -C extra-filename=-841f90be8fa84132 --out-dir /home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps -L dependency=/home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps --extern libc=/home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps/liblibc-c2e5094204db2b1e.rmeta --cap-lints allow --target thumbv7neon-unknown-linux-gnueabihf`
   Compiling serde v1.0.138
     Running `rustc --crate-name build_script_build /home/users/builder/.cargo/registry/src/github.com-1285ae84e5963aae/serde-1.0.138/build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="serde_derive"' --cfg 'feature="std"' -C metadata=d95a6a7c3c54c589 -C extra-filename=-d95a6a7c3c54c589 --out-dir /home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/build/serde-d95a6a7c3c54c589 -L dependency=/home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps --cap-lints allow --target thumbv7neon-unknown-linux-gnueabihf`
     Running `/home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/build/serde-d95a6a7c3c54c589/build-script-build`
   Compiling cc v1.0.66
     Running `rustc --crate-name cc --edition=2018 /home/users/builder/.cargo/registry/src/github.com-1285ae84e5963aae/cc-1.0.66/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="jobserver"' --cfg 'feature="parallel"' -C metadata=6b189a677b327c21 -C extra-filename=-6b189a677b327c21 --out-dir /home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps -L dependency=/home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps --extern jobserver=/home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps/libjobserver-841f90be8fa84132.rmeta --cap-lints allow --target thumbv7neon-unknown-linux-gnueabihf`
     Running `rustc --crate-name memchr --edition=2018 /home/users/builder/.cargo/registry/src/github.com-1285ae84e5963aae/memchr-2.4.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="std"' --cfg 'feature="use_std"' -C metadata=3ed86314f4a405e2 -C extra-filename=-3ed86314f4a405e2 --out-dir /home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps -L dependency=/home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps --cap-lints allow --target thumbv7neon-unknown-linux-gnueabihf`
     Running `rustc --crate-name serde_derive /home/users/builder/.cargo/registry/src/github.com-1285ae84e5963aae/serde_derive-1.0.138/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="default"' -C metadata=348dbf0b7f3e1fab -C extra-filename=-348dbf0b7f3e1fab --out-dir /home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps -L dependency=/home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps --extern proc_macro2=/home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps/libproc_macro2-a9b4b22ce241c217.rlib --extern quote=/home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps/libquote-4542bb92c1a1bf1b.rlib --extern syn=/home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps/libsyn-846b5e648ddc33af.rlib --extern proc_macro --cap-lints allow --target thumbv7neon-unknown-linux-gnueabihf --cfg underscore_consts --cfg ptr_addr_of`
   Compiling pkg-config v0.3.19
     Running `rustc --crate-name pkg_config /home/users/builder/.cargo/registry/src/github.com-1285ae84e5963aae/pkg-config-0.3.19/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 -C metadata=006d0a394f70d84d -C extra-filename=-006d0a394f70d84d --out-dir /home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps -L dependency=/home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps --cap-lints allow --target thumbv7neon-unknown-linux-gnueabihf`
     Running `rustc --crate-name serde /home/users/builder/.cargo/registry/src/github.com-1285ae84e5963aae/serde-1.0.138/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="serde_derive"' --cfg 'feature="std"' -C metadata=4d15dfd3fa26bbb6 -C extra-filename=-4d15dfd3fa26bbb6 --out-dir /home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps -L dependency=/home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps --extern serde_derive=/home/users/builder/rpm/packages/delta/delta-0.15.1/target/debug/deps/libserde_derive-348dbf0b7f3e1fab.so --cap-lints allow --target thumbv7neon-unknown-linux-gnueabihf --cfg no_std_atomic64 --cfg no_std_atomic`
error[E0463]: can't find crate for `serde_derive`
   --> /home/users/builder/.cargo/registry/src/github.com-1285ae84e5963aae/serde-1.0.138/src/lib.rs:294:1
    |
294 | extern crate serde_derive;
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^ can't find crate

error[E0432]: unresolved imports `self::__private`, `self::__private`
   --> /home/users/builder/.cargo/registry/src/github.com-1285ae84e5963aae/serde-1.0.138/src/lib.rs:276:5
    |
276 | use self::__private as export;
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^
277 | #[allow(unused_imports)]
278 | use self::__private as private;
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^

However moving --target from RUSTFLAGS to cargo command line makes build successful:

CC=gcc cargo build --target thumbv7neon-unknown-linux-gnueabihf --verbose -j 1

This seems to affect platforms from tiers without hosts tools only? Managed to reproduce it for thumbv7neon-unknown-linux-gnueabihf and x86_64-unknown-linux-gnux32 but not on x86_64-unknown-linux-gnu, i686-unknown-linux-gnu, aarch64-unknown-linux-gnu, arm-unknown-linux-gnueabihf or armv7-unknown-linux-gnueabihf.

Steps

No response

Possible Solution(s)

No response

Notes

No response

Version

$ cargo version --verbose
cargo 1.65.0
release: 1.65.0
host: armv7-unknown-linux-gnueabihf
libgit2: 1.4.4 (sys:0.15.0 system)
libcurl: 7.86.0 (sys:0.4.55+curl-7.83.1 system ssl:OpenSSL/3.0.7)
os: Linux [32-bit]
weihanglo commented 1 year ago

Cargo does more on its own --target flag than simply passing down to rustc, such as platform dependency activation, feature unification, and build cache management, and probably more. And yeah, it could be missing host tools leading to this issue.

To me, I would avoid passing RUSTFLAGS unless there is no alternative in Cargo. It might mess up rustc invocations and some assumptions from Cargo perhaps.

Out of curious, is there anything hinder you from using cargo --target?

jpalus commented 1 year ago

Out of curious, is there anything hinder you from using cargo --target?

The boring story is:

  1. I'm packaging rust based software for rpm based distribution. Majority of software is obviously cargo based and hence so far I've had %cargo_build macro that included both RUSTFLAGS with some minor flags as well as cargo invocation with --target as cargo argument. Everything worked fine obviously.
  2. Then mesa incorporated rusticl that is not cargo based but instead uses meson. So now I needed to pass --target somewhere else and it seemed the only option was to pass it in RUSTFLAGS. Hence --target became part of RUSTFLAGS as well.
  3. Then I noticed rustc is no longer happy about --target being passed twice, both by RUSTFLAGS and cargo (even though it has the same value, it would be cool if it did not complain in that case): 
    error: Option 'target' given more than once
  4. So i dropped it from cargo to rely on RUSTFLAGS only. And here we are.

I guess it would be possible to provide RUSTFLAGS and build tool arguments for each and every build system out there, but it would be really nice if it wasn't necessary. I wish rustc did not fail for --target passed more than once with same value and it would be awesome if cargo could derive --target from RUSTFLAGS if it's present there and no --target was passed in command line.

weihanglo commented 1 year ago

Thank you for your reply. It is an interesting use case. Let me tell the hard part from Cargo's perspective, and we then try to find a workaround maybe.

if cargo could derive --target from RUSTFLAGS if it's present there and no --target was passed in command line.

Cargo already learns RUSTFLAGS from a variety of sources like cfg(…) and platform triples. It would be more convoluted to converge the final RUSTFLAGS if we go that way. See:

https://github.com/rust-lang/cargo/blob/a4d2e29e213d2a98ce8df56ada20dbe9a7391ae8/src/cargo/core/compiler/build_context/target_info.rs#L271-L279

And if I understand correctly, looks like you're trying to cross-compile from armv7-unknown-linux-gnueabihf to thumbv7neon-unknown-linux-gnueabihf, right? RUSTFLAGS for cargo is applied to every rustc invocation. That means things ought to run on your host platform, like build script and procedual macros, compile to target platform. I guess this might not be the behaviour you desire. I am not familiar with how rusticl handles such tools, though.

Personally, I guess it is more likely to do it in reverse: when RUSTFLAGS contains --target, Cargo warns and ignores it. Maybe filter it from here or elsewhere. However, I believe there are more flags other than --target required to remove from RUSTFLAGS. It is a somewhat breaking change as well.

BTW, if you want the other environment variable to control target platform of Cargo, CARGO_BUILD_TARGET may help.

jpalus commented 1 year ago

And if I understand correctly, looks like you're trying to cross-compile from armv7-unknown-linux-gnueabihf to thumbv7neon-unknown-linux-gnueabihf, right?

Correct. Since thumbv7neon-unknown-linux-gnueabihf (or mentioned x86_64-unknown-linux-gnux32) lacks its own host tools there's no other way but to cross compile (at least as far as I understand rust toolchain).

I am not familiar with how rusticl handles such tools, though.

Just to clarify it's rather how meson handles such tools (meson is used to build rust based rusticl).

Personally, I guess it is more likely to do it in reverse: when RUSTFLAGS contains --target, Cargo warns and ignores it.

Indeed that would work.

BTW, if you want the other environment variable to control target platform of Cargo, CARGO_BUILD_TARGET may help.

Yes I'm aware of it but it suffers from the same issue of defining --target twice. The question really is if it's possible to have common shared RUSTFLAGS between various build systems that contains --target. But turning things around -- couldn't --target in RUSTFLAGS be treated same way as CARGO_BUILD_TARGET but without passing additional --target to rustc?