Local tests fail on x86-64 Linux since rust-lang/cc-rs#991

[workingjubilee]

Bisected a regression in running tests on main to 328f8d83458cbdf7c62469a206bdcf2aebd9eb68

running 40 tests
LLVM (http://llvm.org/):
  LLVM version 17.0.6
  Optimized build.
test gnu_extra_warnings0 ... ok
test compile_intermediates ... ok
test gnu_compile_assembly ... ok
test gnu_define ... ok
test gnu_debug_fp_auto ... ok
test gnu_debug_fp ... ok
test gnu_extra_warnings1 ... ok
cargo:warning=llvm-ar: error: /home/jubilee/rust/cc-rs/target/debug/gcc-test7gypcv/d1fba762150c532c-foo.o: No such file or directory
error: test failed, to rerun pass `--test test`

Caused by:
  process didn't exit successfully: `/home/jubilee/rust/cc-rs/target/debug/deps/test-9add85ae16a35a83` (exit status: 1)

I run x86-64 Linux and cargo test fails here. Not clear why. If the tests now require something be installed that wasn't installed before, that's a documentation bug, I would think.

cc @dpaoliello @NobodyXu

It is possible this may be related to the failure to upgrade cc-rs for rust-lang/rust's bootstrap to 1.0.90 in https://github.com/rust-lang/rust/pull/122504 but I don't know if it is.

[NobodyXu]

That's indeed strange, our CI runs on Linux and it seems to be fine in GHA

[workingjubilee]

It seems the only test failure is in the new test. Got this error after running it again:

cargo:warning=llvm-ar: error: /home/jubilee/rust/cc-rs/target/debug/gcc-test8j2viv/d1fba762150c532c-foo.o: No such file or directory

Then:

$ pwd
/home/jubilee/rust/cc-rs/target/debug/gcc-test8j2viv
$ ls .
arm-linux-androideabi-ar  arm-linux-androideabi-clang  out0
$ file arm-linux-androideabi-ar
arm-linux-androideabi-ar: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=de460af44ed713574f4a99eddcef8b23598e40b5, for GNU/Linux 4.4.0, with debug_info, not stripped
$ file arm-linux-androideabi-clang
arm-linux-androideabi-clang: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=de460af44ed713574f4a99eddcef8b23598e40b5, for GNU/Linux 4.4.0, with debug_info, not stripped
$ ls
arm-linux-androideabi-ar  arm-linux-androideabi-clang  out0
$ cat out0
-O2
-DANDROID
-ffunction-sections
-fdata-sections
-fPIC
-o
/home/jubilee/rust/cc-rs/target/debug/gcc-test8j2viv/d1fba762150c532c-foo.o
-c
foo.c

No idea what I am possibly missing.

[dpaoliello]

We don't shim llvm-ar when running that test on Linux, hence it is running the real llvm-ar which is not finding the .o because the compiler is correctly shimmed.

What I don't understand is why it is trying to use llvm-ar instead of arm-linux-androideabi-ar like the CI machines do - I'm guessing that your machine doesn't have the Android NDK installed, so get_base_archiver_variant initially sets tool_opt to None and so falls back to using llvm-{} for Android: https://github.com/rust-lang/cc-rs/blob/f36d6a7ed06033c38021ef65e2b7f1da38932024/src/lib.rs#L3172-L3176

If so, then the fix would be to shim llvm-ar on Linux as well to handle the case where the Android NDK is missing.

[correabuscar]

and so falls back to using llvm-{} for Android:

confirmed so, on my machine.

[dpaoliello]

I have a potential fix (shimming llvm-ar as well). Can someone seeing this please try it out: https://github.com/rust-lang/cc-rs/pull/1016

[correabuscar]

I have a potential fix (shimming llvm-ar as well). Can someone seeing this please try it out: #1016

now all tests pass on my machine, thanks!

[correabuscar]

Before your patch, I was having trouble finding which was the name of the test that failed, oddly enough(isn't it?) it shows it to me only with 1 test threads:

$ cargo test -- --test-threads 1|grep -i clang_android
    Finished test [unoptimized + debuginfo] target(s) in 0.05s
     Running unittests src/lib.rs (target/debug/deps/cc-29a76a7f882a178a)
     Running unittests src/bin/gcc-shim.rs (target/debug/deps/gcc_shim-4ad2ed7386627e02)
     Running tests/cc_env.rs (target/debug/deps/cc_env-7b57c8d52311f6f6)
     Running tests/cflags.rs (target/debug/deps/cflags-e216f0d13260bc6e)
     Running tests/cxxflags.rs (target/debug/deps/cxxflags-c4d54d1fad6c05cb)
     Running tests/test.rs (target/debug/deps/test-99212c6f8e7cfed2)
test clang_android ... LLVM (http://llvm.org/):
error: test failed, to rerun pass `--test test`

Caused by:
  process didn't exit successfully: `/tmp/1/cc-rs/target/debug/deps/test-99212c6f8e7cfed2 --test-threads 1` (exit status: 1)
note: test exited abnormally; to see the full output pass --nocapture to the harness.

but not with 2 or no test threads specified(presumably this uses 2 or more by default then):

$ cargo test -- --test-threads 2|grep -i clang_android
    Finished test [unoptimized + debuginfo] target(s) in 0.01s
     Running unittests src/lib.rs (target/debug/deps/cc-29a76a7f882a178a)
     Running unittests src/bin/gcc-shim.rs (target/debug/deps/gcc_shim-4ad2ed7386627e02)
     Running tests/cc_env.rs (target/debug/deps/cc_env-7b57c8d52311f6f6)
     Running tests/cflags.rs (target/debug/deps/cflags-e216f0d13260bc6e)
     Running tests/cxxflags.rs (target/debug/deps/cxxflags-c4d54d1fad6c05cb)
     Running tests/test.rs (target/debug/deps/test-99212c6f8e7cfed2)
error: test failed, to rerun pass `--test test`

Caused by:
  process didn't exit successfully: `/tmp/1/cc-rs/target/debug/deps/test-99212c6f8e7cfed2 --test-threads 2` (exit status: 1)
note: test exited abnormally; to see the full output pass --nocapture to the harness.

$ cargo test -- |grep -i clang_android
    Finished test [unoptimized + debuginfo] target(s) in 0.01s
     Running unittests src/lib.rs (target/debug/deps/cc-29a76a7f882a178a)
     Running unittests src/bin/gcc-shim.rs (target/debug/deps/gcc_shim-4ad2ed7386627e02)
     Running tests/cc_env.rs (target/debug/deps/cc_env-7b57c8d52311f6f6)
     Running tests/cflags.rs (target/debug/deps/cflags-e216f0d13260bc6e)
     Running tests/cxxflags.rs (target/debug/deps/cxxflags-c4d54d1fad6c05cb)
     Running tests/test.rs (target/debug/deps/test-99212c6f8e7cfed2)
error: test failed, to rerun pass `--test test`

Caused by:
  process didn't exit successfully: `/tmp/1/cc-rs/target/debug/deps/test-99212c6f8e7cfed2` (exit status: 1)
note: test exited abnormally; to see the full output pass --nocapture to the harness.

Do you think this is a bug in cargo maybe? somehow having some output(such as the (not yet '\n') line test clang_android ...) get lost somewhere?

EDIT: I'm looking into it anyway, it seems to point to libtest from rust code, so far, not cargo per se.

EDIT2: mitigation, for me, is

[env]
RUST_TEST_THREADS = "1"
#^ from: https://github.com/rust-lang/cargo/issues/8430#issuecomment-1146812150

in ~/.cargo/config[.toml] EDIT3: so this happens because the test itself exits the process and libtest of rust thus loses all output from the test because of this. I'm still on it... I'll try to report it upstream.

[correabuscar]

This exit here is the reason why we can't see which test failed and the test harness itself doesn't report it nicely: https://github.com/rust-lang/cc-rs/blob/0195ebfcf070d82fb9ae1e5ac41fe35cb9498d91/src/lib.rs#L3740-L3743

so the current output is (incorrect):

     Running tests/test.rs (target/debug/deps/test-4c20fca8914bddfa)

running 40 tests
LLVM (http://llvm.org/):
  LLVM version 18.1.0
  Optimized build.
test gnu_extra_warnings0 ... ok
test gnu_define ... ok
test gnu_extra_warnings1 ... ok
test gnu_debug_fp_auto ... ok
test gnu_debug_fp ... ok
test gnu_compile_assembly ... ok
test compile_intermediates ... ok
test asm_flags ... ok
cargo:warning=llvm-ar: error: /tmp/2/cc-rs/target/debug/gcc-testpICPy2/d1fba762150c532c-foo.o: No such file or directory
error: test failed, to rerun pass `--test test`

Caused by:
  process didn't exit successfully: `/tmp/2/cc-rs/target/debug/deps/test-4c20fca8914bddfa` (exit status: 1)
note: test exited abnormally; to see the full output pass --nocapture to the harness.

^ where you can't see the test that's causing the problem (clang_android that is),

However output with RUST_TEST_THREADS=1 (or cargo test -- --test-threads=1) is(correct):

     Running tests/test.rs (target/debug/deps/test-4c20fca8914bddfa)

running 40 tests
test asm_flags ... ok
test clang_android ... LLVM (http://llvm.org/):
  LLVM version 18.1.0
  Optimized build.
cargo:warning=llvm-ar: error: /tmp/2/cc-rs/target/debug/gcc-testGkumgx/d1fba762150c532c-foo.o: No such file or directory
error: test failed, to rerun pass `--test test`

Caused by:
  process didn't exit successfully: `/tmp/2/cc-rs/target/debug/deps/test-4c20fca8914bddfa` (exit status: 1)
note: test exited abnormally; to see the full output pass --nocapture to the harness.

if it were a panic then it would looks nicer in the tests, like this:

     Running tests/test.rs (target/debug/deps/test-4c20fca8914bddfa)

running 40 tests
test asm_flags ... ok
test clang_android ... LLVM (http://llvm.org/):
  LLVM version 18.1.0
  Optimized build.
cargo:warning=llvm-ar: error: /tmp/2/cc-rs/target/debug/gcc-testIMqra0/d1fba762150c532c-foo.o: No such file or directory
FAILED
test compile_intermediates ... ok
test gnu_apple_darwin ... ok
test gnu_compile_assembly ... ok
test gnu_debug ... ok
test gnu_debug_fp ... ok
test gnu_debug_fp_auto ... ok
test gnu_debug_nofp ... ok
test gnu_define ... ok
test gnu_extra_warnings0 ... ok
test gnu_extra_warnings1 ... ok
test gnu_flag_if_supported ... ok
test gnu_flag_if_supported_cpp ... ok
test gnu_i686 ... ok
test gnu_i686_pic ... ok
test gnu_include ... ok
test gnu_no_dash_dash ... ok
test gnu_opt_level_1 ... ok
test gnu_opt_level_s ... ok
test gnu_set_stdlib ... ok
test gnu_shared ... ok
test gnu_smoke ... ok
test gnu_static ... ok
test gnu_std_c ... ok
test gnu_warnings ... ok
test gnu_warnings_into_errors ... ok
test gnu_warnings_overridable ... ok
test gnu_x86_64 ... ok
test gnu_x86_64_no_pic ... ok
test gnu_x86_64_no_plt ... ok
test msvc_debug ... ok
test msvc_define ... ok
test msvc_include ... ok
test msvc_no_dash_dash ... ok
test msvc_no_static_crt ... ok
test msvc_opt_level_0 ... ok
test msvc_smoke ... ok
test msvc_static_crt ... ok
test msvc_std_c ... ok

failures:

---- clang_android stdout ----
cargo:rerun-if-env-changed=CC_arm-linux-androideabi
CC_arm-linux-androideabi = None
cargo:rerun-if-env-changed=CC_arm_linux_androideabi
CC_arm_linux_androideabi = None
cargo:rerun-if-env-changed=TARGET_CC
TARGET_CC = None
cargo:rerun-if-env-changed=CC
CC = None
cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
cargo:warning=Compiler family detection failed due to error: ToolNotFound: Failed to find tool. Is `arm-linux-androideabi-clang` installed?
cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
CRATE_CC_NO_DEFAULTS = None
cargo:rerun-if-env-changed=CFLAGS_arm-linux-androideabi
CFLAGS_arm-linux-androideabi = None
cargo:rerun-if-env-changed=CFLAGS_arm_linux_androideabi
CFLAGS_arm_linux_androideabi = None
cargo:rerun-if-env-changed=TARGET_CFLAGS
TARGET_CFLAGS = None
cargo:rerun-if-env-changed=CFLAGS
CFLAGS = None
cargo:rerun-if-env-changed=AR_arm-linux-androideabi
AR_arm-linux-androideabi = None
cargo:rerun-if-env-changed=AR_arm_linux_androideabi
AR_arm_linux_androideabi = None
cargo:rerun-if-env-changed=TARGET_AR
TARGET_AR = None
cargo:rerun-if-env-changed=AR
AR = None
cargo:rerun-if-env-changed=ARFLAGS_arm-linux-androideabi
ARFLAGS_arm-linux-androideabi = None
cargo:rerun-if-env-changed=ARFLAGS_arm_linux_androideabi
ARFLAGS_arm_linux_androideabi = None
cargo:rerun-if-env-changed=TARGET_ARFLAGS
TARGET_ARFLAGS = None
cargo:rerun-if-env-changed=ARFLAGS
ARFLAGS = None
thread 'clang_android' panicked at /tmp/2/cc-rs/src/lib.rs:3741:5:

error occurred: Command GCCTEST_OUT_DIR="/tmp/2/cc-rs/target/debug/gcc-testIMqra0" PATH="/tmp/2/cc-rs/target/debug/gcc-testIMqra0:/swcode/bin.prepend:/home/user/bin/binprio:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/lib/llvm/18/bin:/usr/lib/llvm/17/bin:/home/user/bin:/home/user/bin/oldbin:/opt/depot_tools/:/swcode/bin.append:/home/user/.cargo/bin" ZERO_AR_DATE="1" "llvm-ar" "cq" "/tmp/2/cc-rs/target/debug/gcc-testIMqra0/libfoo.a" "/tmp/2/cc-rs/target/debug/gcc-testIMqra0/d1fba762150c532c-foo.o" with args llvm-ar did not execute successfully (status code exit status: 1).

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

failures:
    clang_android

test result: FAILED. 39 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.37s

error: test failed, to rerun pass `--test test`

diff --git a/src/lib.rs b/src/lib.rs
index b371051..1163b3c 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -3738,8 +3738,8 @@ impl Default for Build {
 }

 fn fail(s: &str) -> ! {
-    eprintln!("\n\nerror occurred: {}\n\n", s);
-    std::process::exit(1);
+    panic!("\n\nerror occurred: {}\n\n", s);
+    //std::process::exit(1);
 }

 #[derive(Clone, Copy, PartialEq)]