Suboptimal niche usage within the fields of a variant.

[eddyb]

On x86_64, this prints the values in comments:

pub enum Option2<A, B> {
    Some(A, B),
    None
}

fn main() {
    use std::mem::size_of;

    dbg!(size_of::<Option<Option<(bool, &())>>>()); // = 16
    dbg!(size_of::<Option<Option<(&(), bool)>>>()); // = 16
    dbg!(size_of::<Option<Option2<bool, &()>>>()); // = 16
    dbg!(size_of::<Option<Option2<&(), bool>>>()); // = 24
}

What's happening here is that when we pick the niche to represent None of Option<T>, we pick the largest (or "roomiest") of the niches in T, so that further enums around Option<T> have more optimization opportunities themselves.

(Why just one niche? That has to do with the fact that in an enum, the only part of the representation guaranteed to be always initialized and valid is the tag / niche, but no other variant fields)

For Option2<A, B>, however, we just pick the best niche in A and only look at B if A has no niches at all, so if B has a better niche, we miss it.

This is a small oversight in the code itself, and arguably it will be slightly nicer once this is fixed, but I wanted to open this issue first, before I have a chance to fix it myself (if someone else doesn't get there first).

cc @Gankra @nox

[nox]

I was pretty sure that my patch to find the roomiest niche covered that case, but apparently it didn't. :(