[CI] Debug failures

[giordano]

@gbaraldi now here I'm getting a new error (https://github.com/JuliaPackaging/BinaryBuilderBase.jl/actions/runs/6754524952/job/18362088675?pr=346#step:8:131):

Compilation - Linux x86_64 {cxxstring_abi=cxx11, libc=musl} - clang++: Test Failed at /home/runner/work/BinaryBuilderBase.jl/BinaryBuilderBase.jl/test/runners.jl:207
  Expression: readchomp(iobuff) == string(cmd)
   Evaluated: "`/bin/bash -c \"set -e\necho '#include <complex>\nstd::complex<double> add(std::complex<double> a, std::complex<double> b) {\n    return a + b;\n}\n' > test.cpp\necho '    #include <complex>\n    std::complex<double> add(std::complex<double> a, std::complex<double> b);\n    int main(void) {\n        std::complex<double> z3 = add(std::complex<double>(1.,2.),std::complex<double>(4.,2.));\n        return 0;\n    }\n' > main.cpp\n# Build object file\nclang++ -Werror -std=c++11 -c test.cpp -o test.o\n# Build shared library\nclang++ -Werror -std=c++11 -fPIC -shared test.cpp -o libtest.\\\${dlext}\n# Build and link program with object file\nclang++ -Werror -std=c++11 -o main main.cpp test.o\n# Build and link program with shared library\nclang++ -Werror -std=c++11 -o main main.cpp -L. -ltest\n\"`\nld.lld: error: /opt/x86_64-linux-musl/lib/gcc/x86_64-linux-musl/5.2.0/../../../../x86_64-linux-musl/lib/../lib64/libstdc++.so: invalid local symbol '_edata' in global part of symbol table\nld.lld: error: /opt/x86_64-linux-musl/lib/gcc/x86_64-linux-musl/5.2.0/../../../../x86_64-linux-musl/lib/../lib64/libstdc++.so: invalid local symbol '_end' in global part of symbol table\nld.lld: error: /opt/x86_64-linux-musl/lib/gcc/x86_64-linux-musl/5.2.0/../../../../x86_64-linux-musl/lib/../lib64/libstdc++.so: invalid local symbol '__bss_start' in global part of symbol table\nclang-16: error: linker command failed with exit code 1 (use -v to see invocation)" == "`/bin/bash -c \"set -e\necho '#include <complex>\nstd::complex<double> add(std::complex<double> a, std::complex<double> b) {\n    return a + b;\n}\n' > test.cpp\necho '    #include <complex>\n    std::complex<double> add(std::complex<double> a, std::complex<double> b);\n    int main(void) {\n        std::complex<double> z3 = add(std::complex<double>(1.,2.),std::complex<double>(4.,2.));\n        return 0;\n    }\n' > main.cpp\n# Build object file\nclang++ -Werror -std=c++11 -c test.cpp -o test.o\n# Build shared library\nclang++ -Werror -std=c++11 -fPIC -shared test.cpp -o libtest.\\\${dlext}\n# Build and link program with object file\nclang++ -Werror -std=c++11 -o main main.cpp test.o\n# Build and link program with shared library\nclang++ -Werror -std=c++11 -o main main.cpp -L. -ltest\n\"`"

It's with clang and it's a linker error involving lld, but the problem is that I can't reproduce it locally. The failing test is https://github.com/JuliaPackaging/BinaryBuilderBase.jl/blob/8d1eca1e3a63a2de30d9a9eee7381c9ca999ea60/test/runners.jl#L207

Edit: ~this still appears after rebasing on top of #345.~ Sorry, I was looking at the wrong log, this goes away with your PR: https://github.com/JuliaPackaging/BinaryBuilderBase.jl/actions/runs/6754635721/job/18362311393?pr=346

[giordano]

Alright, I finally managed to get some more useful error messages, for the failure at https://github.com/JuliaPackaging/BinaryBuilderBase.jl/blob/8d1eca1e3a63a2de30d9a9eee7381c9ca999ea60/test/runners.jl#L446

rustc -o /tmp/testsuite/x86_64-linux-gnu/rust/hello_world/hello_world hello_world.rs
  Installing cargo_build v0.1.0 (/usr/share/testsuite/rust/cargo_build)
error: failed to compile `cargo_build v0.1.0 (/usr/share/testsuite/rust/cargo_build)`, intermediate artifacts can be found at `/usr/share/testsuite/rust/cargo_build/target`.
To reuse those artifacts with a future compilation, set the environment variable `CARGO_TARGET_DIR` to that path.

Caused by:
  Cross-device link (os error 18)
make[1]: *** [Makefile:7: /tmp/testsuite/x86_64-linux-gnu/rust/cargo_build/cargo_build] Error 101
make: *** [Makefile:48: install-project-rust-cargo_build] Error 2

Sounds like a problem in the sandbox code, which perhaps only arises on the CI setup.

[giordano]

My diagnosis, reported in https://github.com/JuliaPackaging/BinaryBuilderBase.jl/pull/325#issuecomment-1793450297, is that we're running out of disk in the non-squashfs jobs, using Julia v1.9 was useful as it prints a more informative error message when a download fails.