Missed niche optimization

[Jules-Bertholet]

I tried this code:

#[repr(u8)]
enum Foo {
    A = 0,
}

#[repr(u8)]
enum Bar {
    A = 1,
}

enum Choice {
    F(Foo),
    B(Bar),
}

fn main() {
    dbg!(core::mem::size_of::<Choice>());
}

I expected to see this happen: Choice has size 1

Instead, this happened: Choice has size 2

Meta

Rust version; 1.74.1

@rustbot label A-layout T-compiler

[programmerjake]

this optimization should apply when variants have data too:

// could be size 2 since A and B's discriminants are in the same spot with non-overlapping contiguous values, but currently isn't
enum Top {
    A(A),
    B(B),
}

#[repr(u8)]
enum A {
    A0(u8) = 0,
    A1(u8) = 1,
}

#[repr(u8)]
enum B {
    B2(u8) = 2,
    B3(u8) = 3,
}