Real cross-compiling tests instead of `#![no_core]` silliness

[workingjubilee]

We have a zillion tests that use the #![no_core] hack, like this one: https://github.com/rust-lang/rust/blob/5e3ede22ef6fd23197629c22b9f466b3734e19f1/tests/ui/asm/bad-template.rs#L10

A given test involves:

• Host A
• Target B
• Some Rust source (compiled for Target B)
• Test tooling (run on Host A)

Most of our test infra assumes that Host A and Target B are the same thing. This greatly simplifies a lot of assumptions. But this hack we have introduced in so many tests essentially exists so that we can be sure the test runs even when cross-compiling from Host A to Target B. It's possible there's some weird finagling of test flags that can enable this without the hack, but that would be another hack itself. Cross-compiled tests should be relatively easy, like just writing something like:

//@ cross-compile-targets: arch_1-unknown-os_3-env_5 arch_2-unknown-os_4-env_6 arch_1-unknown-os_4-env_5 arch_2-unknown-os_3-env_7

[workingjubilee]

sorry @jieyouxu!

[nikic]

The problem here is that without no_core, you actually have to build core/std for the target. We have some tests that test ~all targets, and having everyone build core/std for 100 targets just to run some codegen or assembly tests would be ... not great.

[workingjubilee]

That would probably be bad, yes. Handling this correctly probably requires some coordination between bootstrap and compiletest about which stdlibs are built (or can be built, or may be desirable to build).

[workingjubilee]

I don't know that we'd even need much to extract information like "Ignored 20 assembly tests with 50 revisions because you didn't build std for aarch64-apple-darwin and 69 other targets."

That would presume treating it like something between needs-llvm-component and compile-flags: --target.

It probably wouldn't replace every no_core test. But there are many cases where no_core tests aren't good enough or are too tedious to write so they simply don't get written, and then there's no way to run the test that didn't get written, but if we had at least the core built we could write the tests.

[RalfJung]

We also have the problem that for most of our targets, you can't easily cross-compile the sysroot. Building a windows std on my Linux host doesn't "just work", at the very least it requires a bunch of tooling to be installed system-wide (because std has C dependencies, and because a linker is needed to build the final artifact). As long as that requirement stands, I think the no_core approach is crucial to be able to let people run and --bless those tests without having to install all sorts of stuff on their system.

We don't actually need std for these tests, though. Can we build a "core-only sysroot" for these targets when needed? That should be faster than building a full sysroot, and it should in principle be possible without any target-specific tools.

That said, looking at one of my own no_core tests (tests/ui/abi/compatibility.rs), I definitely don't want to have to build core 12 times just to be able to run that test on all targets. So I think we want to keep tests like this no_core. I don't think that's actually such a bad hack, TBH -- the only thing that bothers me about it is having to update all those lang items in a bunch of tests. That can be fixed with a minicore.rs file that we include! into these no_core tests, rather than having to copy-paste the same stuff everywhere.

[chrisnc]

+1 for a minicore.rs with all the lang_items defined. As I was writing out a half-dozen lang items trying to get the arm fpu feature test to work with #![no_core], this is exactly what I was wishing for, as nothing about them was specific to the test I was trying to write. I think I would have needed about a half dozen more to get the test to work but couldn't figure out how to use some of them and failed at finding examples to follow.

[workingjubilee]

I'd be satisfied-enough with that, as long as we can cut down the silliness!

[jieyouxu]

@RalfJung so basically, pull out this incantation into a separate "header" file that would be made available to tests that need #![no_core]?

#[cfg(host)]
use std::{
    any::Any, marker::PhantomData, mem::ManuallyDrop, num::NonZero, ptr::NonNull, rc::Rc, sync::Arc,
};

/// To work cross-target this test must be no_core.
/// This little prelude supplies what we need.
#[cfg(not(host))]
mod prelude {
    #[lang = "sized"]
    pub trait Sized {}

    #[lang = "receiver"]
    pub trait Receiver {}
    impl<T: ?Sized> Receiver for &T {}
    impl<T: ?Sized> Receiver for &mut T {}

    #[lang = "copy"]
    pub trait Copy: Sized {}
    impl Copy for i32 {}
    impl Copy for f32 {}
    impl<T: ?Sized> Copy for &T {}
    impl<T: ?Sized> Copy for *const T {}
    impl<T: ?Sized> Copy for *mut T {}

    #[lang = "clone"]
    pub trait Clone: Sized {
        fn clone(&self) -> Self;
    }

    #[lang = "phantom_data"]
    pub struct PhantomData<T: ?Sized>;
    impl<T: ?Sized> Copy for PhantomData<T> {}

    #[lang = "unsafe_cell"]
    #[repr(transparent)]
    pub struct UnsafeCell<T: ?Sized> {
        value: T,
    }

    pub trait Any: 'static {}

    pub enum Option<T> {
        None,
        Some(T),
    }
    impl<T: Copy> Copy for Option<T> {}

    pub enum Result<T, E> {
        Ok(T),
        Err(E),
    }
    impl<T: Copy, E: Copy> Copy for Result<T, E> {}

    #[lang = "manually_drop"]
    #[repr(transparent)]
    pub struct ManuallyDrop<T: ?Sized> {
        value: T,
    }
    impl<T: Copy + ?Sized> Copy for ManuallyDrop<T> {}

    #[repr(transparent)]
    #[rustc_layout_scalar_valid_range_start(1)]
    #[rustc_nonnull_optimization_guaranteed]
    pub struct NonNull<T: ?Sized> {
        pointer: *const T,
    }
    impl<T: ?Sized> Copy for NonNull<T> {}

    #[repr(transparent)]
    #[rustc_layout_scalar_valid_range_start(1)]
    #[rustc_nonnull_optimization_guaranteed]
    pub struct NonZero<T>(T);

    // This just stands in for a non-trivial type.
    pub struct Vec<T> {
        ptr: NonNull<T>,
        cap: usize,
        len: usize,
    }

    pub struct Unique<T: ?Sized> {
        pub pointer: NonNull<T>,
        pub _marker: PhantomData<T>,
    }

    #[lang = "global_alloc_ty"]
    pub struct Global;

    #[lang = "owned_box"]
    pub struct Box<T: ?Sized, A = Global>(Unique<T>, A);

    #[repr(C)]
    struct RcBox<T: ?Sized> {
        strong: UnsafeCell<usize>,
        weak: UnsafeCell<usize>,
        value: T,
    }
    pub struct Rc<T: ?Sized, A = Global> {
        ptr: NonNull<RcBox<T>>,
        phantom: PhantomData<RcBox<T>>,
        alloc: A,
    }

    #[repr(C, align(8))]
    struct AtomicUsize(usize);
    #[repr(C)]
    struct ArcInner<T: ?Sized> {
        strong: AtomicUsize,
        weak: AtomicUsize,
        data: T,
    }
    pub struct Arc<T: ?Sized, A = Global> {
        ptr: NonNull<ArcInner<T>>,
        phantom: PhantomData<ArcInner<T>>,
        alloc: A,
    }
}
#[cfg(not(host))]
use prelude::*;

[jieyouxu]

I can work on this, I'm just not sure what the exact design should be.

[jieyouxu]

As @compiler-errors mentioned in https://github.com/rust-lang/rust/pull/130466#discussion_r1763281000, we can probably have something like //@ aux-build: minicore.rs but a separate compiletest flag since we don't want to have a copy of minicore.rs under a auxiliary folder relative to every test that needs it.

[RalfJung]

Yeah I am not sure what is the best way to do this -- something like tests/util/minicore.rs that we can include! everywhere would work, but maybe an aux-build is better?

Maybe this should be an MCP, then more t-compiler people can weigh in.

[jieyouxu]

Yeah, I can file a new MCP ~tomorrow so more compiler people can advice.

[jieyouxu]

Yeah I am not sure what is the best way to do this -- something like tests/util/minicore.rs that we can include! everywhere would work, but maybe an aux-build is better?

ui tests have the precedent of a C test_helpers.c file, so I suppose assembly and codegen tests could get their own special support file lol. Just don't want any hard-to-debug cursedness due to staging and whatever.

[jieyouxu]

I have a prototype PR at #130693 and opened MCP at https://github.com/rust-lang/compiler-team/issues/786.