Tracking issue for RFC 2645, "Transparent Unions" (formerly: and Enums)

[Centril]

This is a tracking issue for the RFC "Transparent Unions and Enums" (rust-lang/rfcs#2645).

Steps:

• [x] Implement the RFC (cc @rust-lang/compiler -- can anyone write up mentoring instructions?)
• [ ] Adjust documentation (see instructions on forge)
• [ ] Stabilization PR (see instructions on forge) [done for enums]

Unresolved questions:

The role of #[repr(transparent)] in nonnull-style optimizations is not entirely clear. Specifically, it is unclear whether the user can rely on these optimizations to be performed when they make a type transparent. Transparent unions somewhat complicate the matter. General concensus seems to be that the compiler is free to decide where and when to perform nonnull-style optimizations on unions (regardless of whether or not the union is transaprent), and no guarantees are made to the user about when and if those optimizations will be applied. It is still an open question exactly what guarantees (if any) Rust makes about transparent structs (and enums) and nonnull-style optimizations.

This RFC doesn't propose any changes to transparent structs, and so does not strictly depend on this question being resolved. But since this RFC is attempting to round out the #[repr(transparent)] feature, it seems reasonable to dedicate some time to attempting to round out the guarantees about #[repr(transparent)] on structs.

Also it is not clear if transparent unions can even be implemented on LLVM without seriously restricting our semantics for unions overall.

[mjbshaw]

I plan on attempting to implement the feature by adapting my previous implementation. I'd appreciate it if someone could take a look at that commit and let me know if it's horribly wrong :)

[Centril]

@pnkfelix Up for reviewing ^--- ?

[Centril]

Implementation of both feature gates was done in https://github.com/rust-lang/rust/pull/60463 which landed on 2019-06-11. The PR was written by @mjbshaw and reviewed by @varkor and @rkruppe.

[Centril]

Should we apply #[repr(transparent)] to MaybeUninit now?

[mjbshaw]

I'll submit a PR to update MaybeUninit.

[gnzlbg]

I ran into this today:

#[repr(C)] struct S(u8, u32);

#[repr(transparent)]
union U { _x: S, y: () }

unsafe extern "C" fn foo(u: U) {
    let pad = (&u as *const _ as *const u8).add(1).read();
    assert_eq!(pad, 13); // Should fail when called below
}

pub unsafe fn bar() {
  let mut u: U = U { y: () };
  (&mut u as *mut _ as *mut u8).add(1).write(42);
  foo(u);
}

Because of (), all bit-patterns of union U are valid. The particular bit-pattern being constructed in bar is [undef, 42, undef, undef, undef, undef, undef, undef]. I expected the assert in foo to fail when called from bar: that bit-pattern is a valid bit-pattern for U, foo reads its first byte, and the test always returns false (42 != 13).

However, due to repr(transparent), this is not correct. repr(transparent) gives U the same ABI as S. That is, U will be passed by value "as if" it were an S. The byte of S at offset 1 is padding, and it does not need to be passed when calling foo.

For example, the ABI could state that S.0 is passed in a 1-byte register (e.g. al), and that S.1 is passed in a 32-bit register (e.g. ecx). This is not exactly, but close to what happens when that snippet is compiled for, e.g., i686-pc-windows-msvc (godbolt). The above is compiled in debug mode to:

core::ptr::write:
        ;; writes 42 to the address at ecx
        mov     byte ptr [ecx], 42
        ret
core::ptr::<impl *mut T>::add:
        ;; increments the address at ecx by 1
        lea     eax, [ecx + 1]
        ret
example::bar:
        sub     esp, 24
        ;; our union is at esp + 8, create ptr:
        lea     ecx, [esp + 8]
        ;; increment its address by 1
        call    core::ptr::<impl *mut T>::add
        mov     ecx, eax
        ;; writes 42 to the address (esp + 9)
        call    core::ptr::<impl *mut T>::write
        ;; now passing the union to foo begins
        ;; S.0 is put in `al` - padding be damned
        mov     al, byte ptr [esp + 8]
        ;; S.1 is put in `ecx`
        mov     ecx, dword ptr [esp + 12]
        ;; then these are moved back to the stack (esp + 16) 
        mov     byte ptr [esp + 16], al
        ;; the padding at esp +17/18/19 is uninitialized
        mov     dword ptr [esp + 20], ecx
        ;; and then moved back and forth a couple of times
        movsd   xmm0, qword ptr [esp + 16]
        movsd   qword ptr [esp], xmm0
        ;; and then we call foo, which never sees 42
        call    example::foo
        add     esp, 24
        ret

If this happens, then the bytes at offset 1,2, and 3 of the union U will not be passed to foo, and will become uninitialized. This would allow the compiler to optimize the assert_eq! inside foo away. This is exactly what happens if we crank-up the opt-level to 3 (godbolt):

example::bar:
        ret

If the union U has the same call ABI as S, then the bytes 1, 2, and 3 cannot be part of the "value representation" (cc @ubsan) of the union - they do not need to be memcpy'ed on move, etc. If these bytes need to be part of the "value representation" of the union, then the union cannot have the same call ABI as S - instead, the union would need the same ABI as, e.g., #[repr(C)] struct(i32,i32) so that those bytes get passed by value on copy/move/etc.

AFAICT, these two constraints are incompatible.

cc @eddyb @rust-lang/wg-unsafe-code-guidelines

[RalfJung]

Ouch. Nice catch!

The reason we implemented a "transparent" repr for unions (re-using the ABI of the only non-ZST field if possible) was because of code that used MybeUninit<SimdType> getting a lot slower compared to using mem::uninitialized, where there was no "wrapper". We had to switch MaybeUninit to use the SIMD ABI in that case to get the old performance back. At least that was our interpretation back then, I remember we did more than one thing to fix the perf problems, and I don't remember if we confirmed which one was "it". Here's the relevant PR.

So, a way to keep that while also achieving that "unions preserve all bytes when copied around" would be to only use the ABI of the only non-ZST field if that ABI is an "Integer" or "vector" one, where we know all bytes are preserved.

That would not really be repr(transparent) any more, that flag would have to be removed.

[mjbshaw]

@gnzlbg I think that's UB, with or without repr(transparent). With repr(transparent). Without repr(transparent). The end results are exactly the same (interestingly, repr(C) is different).

I'm not aware of any guarantees or requirements that a wrapper type (whether it's a union or a struct) preserves padding bytes, or that reading padding bytes is valid/safe. If Rust makes any guarantees about that I'd love to know where I can read about them.

[hanna-kruppe]

@mjbshaw Obviously the codegen behavior is the same with and without repr(transparent). You should know, since you implemented the attribute without modifying any part of codegen. This is an issue not just with the documented semantics of repr(transparent), but also with the implementation detail that coincides with the codegen effect of repr(transparent).

I'm not aware of any guarantees or requirements that a wrapper type (whether it's a union or a struct) preserves padding bytes, or that reading padding bytes is valid/safe. If Rust makes any guarantees about that I'd love to know where I can read about them.

Union semantics are not yet nailed down, but as discussed in https://github.com/rust-lang/unsafe-code-guidelines/issues/73, unions are not simply "one of the variants" and indeed there are strong arguments for making unions just "bags of bits" without any special padding, niches, or other special compiler-known restrictions on what they can contain or how their memory can be used. But evidently that's incompatible with the concept of transparent unions.

[mjbshaw]

@rkruppe Part of my comment was in response to this:

I expected the assert in foo to fail when called from bar: that bit-pattern is a valid bit-pattern for U, foo reads its first byte, and the test always returns false (42 != 13).

However, due to repr(transparent), this is not correct.

I don't think repr(transparent) is to blame here and it's a red herring.

Thanks for the link to rust-lang/unsafe-code-guidelines#73 (I've read it in the past but hadn't seen the new comments from the past two weeks). But I'm still not seeing the relationship to padding bytes (particularly, the preservation of padding bytes). I think I'm just missing a piece of context regarding preserving the value of padding bytes (is it because people were thinking of using padding bytes to store extra information and enable layout optimizations? Or something else?).

[gnzlbg]

@mjbshaw

I'm not aware of any guarantees or requirements that a wrapper type (whether it's a union or a struct) preserves padding bytes, [...] Thanks for the link to rust-lang/unsafe-code-guidelines#73 (I've read it in the past but hadn't seen the new comments from the past two weeks). But I'm still not seeing the relationship to padding bytes (particularly, the preservation of padding bytes).

You are assuming that unions have padding.

While this might be by necessity the case for repr(transparent) unions, that isn't necessarily the case for repr(Rust) or repr(C) unions, which some would like to define as having no padding.

[gnzlbg]

If our repr(C) union implementation does not preserve the content of union bytes that are field padding, then it would have a bug (EDIT: AFAIK, it does not).

Per C:

• 6.5.8p5: all union fields are at the same offset and this offset is 0, so there is no padding at the front of an union, or between the union fields
• 6.7.2.1p17 a union can have trailing padding
• Note 51: union padding does not need to be copied on assignment
• 6.2.6.1p6: the fields of an union can have padding (but nothing there says that this padding somehow propagates to the union).

That is, If union U { S s; } has the same size as S, that union has no padding, even though S might have padding. If the union U is larger than S, it has trailing padding which does not need to be copied (per Note 51).

However, C++ is more strict than C here: class.copy.assign.13, which states that the object representation of unions is copied - that is, if the union has trailing padding, C++ requires this trailing padding to be copied on assignment, move, copy, etc. (e.g. see clang in action: https://rust.godbolt.org/z/wqXZXB).

---

The "bag of bits" proposal for repr(Rust) unions gives them the same guarantees than in C++: the content of padding is preserved. This cannot, in general, be satisfied by a #[repr(transparent)] union U<T> { x: T }.

[mjbshaw]

Thanks, @gnzlbg. My remaining question is why do we care about preserving the value of padding bits? Who is going to be observing padding bits, and what are they doing with the values they observe?

I'll attempt to answer my own questions in an attempt to better understand the problem, but please correct me if I'm wrong. I assume the reason people are interested in the padding bits of a union is because of code like this (I'll assume tuples use a C representation for brevity and simplicity):

union Mix {
  f1: (bool, u32),
  f2: (u32, bool),
}
let m = Mix { f1: (false, 3) };
m.f2.0 = 3;

In this case, f1 and f2 each have three padding bytes (though in different places) and m is storing two u32 values. When m is passed by value, both u32 values should be preserved, which means that the padding bytes of f1 and f2 need to be preserved.

Assuming that this is why people care about preserving the padding bits in a union, I would like to propose an alternative view that more naturally resolves the whole issue (at least from my perspective). To use C++'s parlance of "value representation", the "value representation" of a union is the (mathematical) union of the bits that comprise the value representation of each of the union's fields (or put another way: the padding bits in the union are the intersection of the padding bits of its fields). The padding bits in a union don't need to be preserved.

Using this viewpoint, both the Mix example above and the repr(transparent) issue in general are solved. Mix has no padding bits, so both u32 values would be preserved when passing Mix around. U (the union from earlier) does have padding bits, though, and those bits don't need to be preserved. Similarly, union Mix2 { f: (bool, u32) } also has padding bits.

[gnzlbg]

When m is passed by value, both u32 values should be preserved, which means that the padding bytes of f1 and f2 need to be preserved.

This is correct, but I think you might still be confusing the padding of the union with the padding of the union fields. As you mentioned, both (u32, bool) and (bool, u32) have padding, but this does not mean that the union Mix has padding. Inversely, if Mix had padding, that padding might have nothing to do with the padding of its fields, e.g., #[repr(align(16))] union MixA { x: (u32, bool), y: (bool, u32) } might have 8 bytes of trailing padding, but that's independent of whether the fields themselves have padding.

Assuming that this is why people care about preserving the padding bits in a union, I would like to propose an alternative view that more naturally resolves the whole issue (at least from my perspective). To use C++'s parlance of "value representation", the "value representation" of a union is the (mathematical) union of the bits that comprise the value representation of each of the union's fields (or put another way: the padding bits in the union are the intersection of the padding bits of its fields). The padding bits in a union don't need to be preserved.

The main advantage of unions without padding bits is that users can just treat them as a bag of bits. It's simple to specify, teach, etc.

Your proposal would need to include, e.g., the effects of repr(align) on the padding of the union, but it is IMO still quite simple. It cannot apply to repr(C) unions, and it must apply to repr(transparent) unions for these to have the same layout as the type being wrapped.

So that leaves only repr(Rust) unions for having a choice. I can't think of any use case of repr(Rust) unions that would be complicated by giving these unions padding. Users that are writing to the raw union bits directly, e.g., via a pointer to the union, probably have to use repr(C) or repr(transparent) anyways. Those using repr(Rust) unions are kind of restricted at modificating the union using field access. Are there any use cases of repr(Rust) unions that would be hurt by giving them padding?

Optimization-wise, this would allow us to not copy padding for repr(Rust) unions. Whether this is worth it, I don't know. My gut tells me that, in most cases, it probably isn't and the backend is going to end up copying the whole thing anyways. Still, we can always implement this as "padding is always copied", and AFAICT, this means that we can change the layout to "unions have no padding" in a backward-compatible way if this turns out to be a bad idea, which is nice. Does giving padding to unions enable any other optimizations?

[RalfJung]

To give another answer to the "why preserve padding" question: The usual way to specify that padding is not preserved when copying structs is that this copies the value representation of the struct. So if we have #[repr(C)] struct S { x: u8, y: u16 }, the process of copying that looks as follows:

• Look at the source bytes ("object representation"), say 0x10 0xFF 0x00 0x40.
• Apply the value representation function. In this case, that returns something like (32, 128). This is the "abstract value" stored in memory: a struct where the first field has value 32 and the second field has value 128. I am using hex for low-level and decimal for high-level values here. (I might have mixed up little endian and big endian here, whatever.) Note in particular that the 2nd byte, 0xFF, has no influence on what happens here.
• Pick some object representation of this value, and put that inpo the target memory. For example, we might pick 0x10 U 0x00 0x40. Here, U is an uninitialized byte (aka undef in LLVM, aka padding byte). This is not the same sequence of bytes as we started with, but that's okay because copying through assignment is a typed operation and we only need to preserve what matters for that type!

So, now what do we do for unions? I don't know what C does (probably something awful involving the "active variant"), but our plan for Rust was to say "the value representation of a union is the same as its object representation". That's the slightly-more-formal version of "unions are just bags of bits". But that would mean that copying at union type, and in particular passing around data at union type, has to preserve the entire object representation!

@gnzlbg

If our repr(C) union implementation does not preserve the content of union bytes that are field padding, then it would have a bug (EDIT: AFAIK, it does not).

"it does not have a bug" or "it does not preserve the bytes"?

However, C++ is more strict than C here: class.copy.assign.13, which states that the object representation of unions is copied - that is, if the union has trailing padding, C++ requires this trailing padding to be copied on assignment, move, copy, etc. (e.g. see clang in action: https://rust.godbolt.org/z/wqXZXB).

That's good news though! It means that if clang is conforming, we can re-use the same machinery from Rust to make sure that all bytes are preserved?

But I don't see how that is encoded. The internet says that LLVM has no unions, just structs, and the LLVM IR for the types in your example look like

%struct.S = type { i8, i32 }
%union.U = type { %struct.S }

Is that enough? You mentioned that rustc doesn't do the right thing even for repr(C), so what are we doing differently?

[RalfJung]

To use C++'s parlance of "value representation", the "value representation" of a union is the (mathematical) union of the bits that comprise the value representation of each of the union's fields (or put another way: the padding bits in the union are the intersection of the padding bits of its fields)

I don't know what that means. "value representations" don't have bits. That's their entire point.

[gnzlbg]

"it does not have a bug" or "it does not preserve the bytes"?

I meant that it does not have a bug.

? You mentioned that rustc doesn't do the right thing even for repr(C), so what are we doing differently?

That is, for repr(C), AFAICT, we are doing the right thing (at least according to the minimal testing I did before).

[eddyb]

The immediate fix, AFAICT, is to not preserve Abi::ScalarPair for unions, only Abi::Scalar and Abi::Vector, from the only non-ZST field. That is, turn Abi::ScalarPair into Abi::Aggregate. (Abi::ScalarPair is a Rust-specific special-case and it's what you get from (u8, u32) - even #[repr(C)] struct S(u8, u32); would get it - and FFI calling conventions treats that as an aggregate)

Neither scalars nor vectors, AFAIK, have any unused or invalid bit patterns (after we reset the validity ranges, which we do for unions anyway). The only exception I can think of is maybe trailing padding from #[repr(align(...))]? (Or trailing padding in weirdly shaped vectors?)

Either way, we can come up with some predicate along the lines of "the size exactly matches all the used & validity-irrelevant bits" for picking scalar/vector (or even "scalar pair") instead of aggregate, if we need to. In this case I think it would amount to "scalar elements fill the entire union size, with no padding".

What I'm more worried about is somehow signalling whether #[repr(transparent)] can do its job. (Do we want to error if there's padding that prevents proper transparency?)

[gnzlbg]

Or trailing padding in weirdly shaped vectors?

We don't support those (yet), right now #[repr(simd)] types need to have a power-of-two number of elements, and #[repr(align, simd)] is not supported. We intend to actually require / guarantee this as part of the UCGs, e.g., see: https://github.com/rust-lang/unsafe-code-guidelines/blob/master/reference/src/layout/packed-simd-vectors.md

What I'm more worried about is somehow signalling whether #[repr(transparent)] can do its job. (Do we want to error if there's padding that prevents proper transparency?)

If that somehow happens, we should definitely bug! out, but that would necessarily have to be a monomorphization-time error, and I'd prefer to come up with a solution that guarantees that #[repr(transparent)] for unions always works (even if that means that whether they can have padding while repr(Rust) unions might not).

[eddyb]

Can we just disallow #[repr(align(N))] on #[repr(transparent)] unions? That should ensure there's no padding around a scalar/vector (and anything else is effectively aggregate).

[gnzlbg]

Can we just disallow #[repr(align(N))] on #[repr(transparent)] unions?

This is not allowed.

[RalfJung]

Supporting padding at the end of a union also wouldn't be too bad. We can just say that unions are "bags of bits" but that the bags might be a bit shorter than size_of::<Union>() because of padding at the end.

The immediate fix, AFAICT, is to not preserve Abi::ScalarPair for unions, only Abi::Scalar and Abi::Vector, from the only non-ZST field. That is, turn Abi::ScalarPair into Abi::Aggregate. (Abi::ScalarPair is a Rust-specific special-case and it's what you get from (u8, u32) - even #[repr(C)] struct S(u8, u32); would get it - and FFI calling conventions treats that as an aggregate)

Agreed.

What I'm more worried about is somehow signalling whether #[repr(transparent)] can do its job.

It would have to statically check "does that field have Scalar or Vector ABI". That seems like such a niche thing that my feeling is it is better to give up on repr(transparent) for unions for now.

[eddyb]

@RalfJung If we always preserve scalar/vector (i.e. without checking for extra padding etc.) then #[repr(transparent)] "just works": the remaining case is aggregates, which an union is itself (and I believe no calling convention distinguishes between an union with one aggregate field + any number of ZST fields and the aggregate field itself - similar thing happens for struct newtypes).

[RalfJung]

How does it "just work"? It currently "just works" and you are proposing to change behavior.

[eddyb]

"Scalar pairs" are indistinguishable from the general aggregate case for non-Rust calling conventions, so making only that change won't break #[repr(transparent)].

But if we start allowing e.g. u8x15 vectors that are 16 bytes long when you include padding, and make unions around u8x15 non-vectors (on account of the one byte of padding), you have a problem - especially with a generic union like MaybeUninit.

According to @gnzlbg, however, none of the edge cases I could think of are allowed, so we can continue to allow scalars and vectors without further checks, thereby maintaining transparency even for generic enums.

[RalfJung]

I am very confused now. @gnzlbg observed padding not being preserved with a repr(Rust) union (that was effectively repr(transparent). So somewhere something has to change. And now @eddyb is arguing that fixing this problem here causes no observable change? How is that possible? Why is the issue not closed yet then?^^

"Scalar pairs" are indistinguishable from the general aggregate case for non-Rust calling conventions

Whether padding is preserved when passing data around by-value using the Rust calling convention is observable to unsafe code.

[gnzlbg]

@RalfJung If an union contains a single scalar, it has the scalar ABI, and has no padding. If a union contains a single Vector, it has the vector ABI, and has no padding. If a union contains a scalar pair, currently, it has the ScalarPair ABI, which has padding, which doesn't get copied. If we remove this, the only remaining case is unions that are Aggregates. These unions have no padding before or in between the fields (all fields are at offset zero), but can have padding after the fields. I'm not sure what the proposed fix for these is.

[RalfJung]

So ScalarPair is where things go wrong currently?

But didn't you use ctest with a non-Rust calling convention, so according to @eddyb that shouldn't be any different from Aggregate?

Maybe we should just tell LLVM that the union is a byte array of the size we computed, and do everything else ourselves.^^

[eddyb]

I'll have to take a closer look at the ctest mentioned, but ScalarPair will definitely drop padding bytes between its A and B scalar elements (and after B), as it behaves as (A, B), outside of FFI.

Note that even with FFI, on the Rust side a ScalarPair will be passed around as its component scalars, so you can't even do *ptr = c_function(*ptr) without losing padding bytes. So maybe that's what the test is exposing? (Even if the FFI treatment is otherwise correct)

[comex]

I'm a little confused by this discussion, so just to be extra clear:

• If you have a struct with padding bytes, there are cases in existing C ABIs where passing or returning that struct to/from functions does not preserve the padding bytes – such as @gnzlbg's example.
• A #[repr(transparent)] union is required be ABI-compatible with the underlying non-zero-sized type, so e.g. MaybeUninit<S> cannot be guaranteed to preserve the padding bytes of S (as opposed to padding bytes of the MaybeUninit itself), at least not when it is passed by value through foreign code.
• This fundamentally requires some sort of specification fix: either reducing the guarantees about unions (all or only #[repr(transparent)]), removing #[repr(transparent)] unions, or making #[repr(transparent)] no longer require ABI compatibility (terrible idea).
• This is separate from any Rust-calling-convention-specific issue.

[eddyb]

@comex Because of how the scalar pair optimization works, you need at least 3 scalar components to actually turn off the conversion between the in-memory form and two SSA values, even around an FFI call, otherwise any test results could still be caused solely by it.

As for your first point, I believe x84_64 SysV (used everywhere other than Windows) might do this, but doesn't it also do the same thing for (non-transparent) unions?

[mjbshaw]

@RalfJung

I don't know what that means. "value representations" don't have bits. That's their entire point.

I'm going to quote the C++ standard, from which I was borrowing the term:

The value representation of an object of type T is the set of bits that participate in representing a value of type T.

This is what I meant by "value representation". It most definitely includes bits. This is the 0x10 _ 0x00 0x40 bit pattern in your example.

So, now what do we do for unions? I don't know what C does (probably something awful involving the "active variant"), but our plan for Rust was to say "the value representation of a union is the same as its object representation".

Thank you. I have a follow-up question: why is the plan for Rust to say "the value representation of a union is the same as its object representation"? What problems does this solve? What advantages does it have?

I'm sorry for my many questions. I'm just trying to understand this thoroughly. So far the clearest answer I've gotten to these questions is "[it's] simple to specify, teach, etc." I agree it's simple to specify, but I'm not convinced it's simple to teach*, nor have I seen any concrete problems that this would solve.

_{*I have some counterpoints that I think help show this, but I'm waiting to post them because I want to make sure I'm arguing the correct points. It wouldn't do this conversation any good if I tried to argue against this without first properly understanding the motivations.}

[eddyb]

@comex @RalfJung I have confirmation that this version of @gnzlbg's test passes with just scalar-pairs disabled and no other changes (just opened the PR to fix this, #62298).

I've talked to @gnzlbg on Discord and I'm pretty clear to me now that the confusion was caused by the rather odd codegen happening, which roughly looked like this (if you'll excuse my pseudocode):

fn bar(u: ptr /* 8 byte stack copy */) {
    pad = load u8, u + 1;
    // assert_eq!(pad, 13);
}
fn foo() {
    u = stack_alloc 8;
    store u8 42, u + 1;
    arg = stack_alloc 8;
    store u8 (load u8, u + 0), arg + 0;
    store u32 (load u32, u + 4), arg + 4;
    bar(arg);
}

So while bar gets a full 8 bytes, creating that arg temporary for passing those 8 bytes on the stack is done by first loading u, and scalar pairs are loaded by each of their two scalar elements. The calling convention treats the union opaquely, but the Rust codegen around the call drops the padding.

[comex]

As for your first point, I believe x84_64 SysV (used everywhere other than Windows) might do this, but doesn't it also do the same thing for (non-transparent) unions?

I checked, and indeed it does. The implication is that C and C++ do not guarantee that padding within structs within unions is preserved when copying the union. I think that contradicts what @gnzlbg said previously, but I could be misunderstanding.

[gnzlbg]

C++ does guarantee that padding is copied when copying an union, the standard is clear class.copy.assign.13:

The implicitly-defined copy assignment operator for a union Xcopies the object representation (basic.types) of X.

Where the object representation is defined as basic.types.4:

The object representation of an object of type T is the sequence of N unsigned char objects taken up by the object of type T, where N equals sizeof(T).

That is, unless the user provides an user-defined copy assignment operator that does something else, the default one provided by the compiler must copy all bytes of the type.

[eddyb]

That is, unless the user provides an user-defined copy assignment operator that does something else, the default one provided by the compiler must copy all bytes of the type.

On Discord we found that at least with the x86_64 SysV ABI, both GCC and clang break that.

[gnzlbg]

https://github.com/rust-lang/rust/issues/60405#issuecomment-507645614

So that's incomplete and incorrect. The standard also says class.union.1:

In a union, a non-static data member is active if its name refers to an object whose lifetime has begun and has not ended ([basic.life]()). At most one of the non-static data members of an object of union type can be active at any time, that is, the value of at most one of the non-static data members can be stored in a union at any time.

That is, if the union has no active member, the lifetime of all objects in the union has ended. If the union has one active member, the lifetime of all other objects in the union has ended. When the lifetime of an object starts, that object is initialized. The contents of padding bytes of an object is undefined.

So essentially, only the value representation of the active field of an union is initialized. All other bytes are uninitialized.

For an union like struct S { char, int32_t } union U { S s } either field S is active or no field is active, in both cases, the union bytes that overlap with the padding bytes of S are uninitialized.

[eddyb]

While trying to generalize the test in #62298 I realized I'd be hitting the issue that there are calling conventions which drop padding bytes from unions, so I ended up with this test that prints all offenders. The lines with "error:" are panics in #62298, and none of them are triggered after my changes.

I can't seem to cause the behavior from https://github.com/rust-lang/rust/issues/60405#issuecomment-507656702 with rustc, as it defaults to {i64, i64} when passing an argument in two registers (instead of clang's i8, i64), so the only types that need to have the extern "C' check demoted to a warning are those where (8 bytes or more of) trailing padding is created with Align16 (as only the first 8 bytes get passed, in one register).

[RalfJung]

@eddyb here is a version that also checks some triples. They seem to all work fine though.

@mjbshaw

Thank you. I have a follow-up question: why is the plan for Rust to say "the value representation of a union is the same as its object representation"? What problems does this solve? What advantages does it have?

It's the only definition I have seen so far that I actually know how to make precise, and that is not extremely complicated. I also think it is simple to teach, and I don't know any other definition that I would dare to even try teaching. Do you have a proposal for another one?

Also to me this is the most natural definition of union: unions are a thing that makes transmute easier to write, by giving you names (the fields of the union) for each type you want to transmute the storage to. Because data gets interpreted only on reads of a field, it makes no sense to make any attempt to interpret data when the entire union is read at union type, so -- just like specified in C++ -- that should preserve all object representation bytes.

I'm sorry for my many questions. I'm just trying to understand this thoroughly.

It's okay, these are good questions and I appreciate the discussion. :)

@eddyb, @gnzlbg

On Discord we found that at least with the x86_64 SysV ABI, both GCC and clang break that.

So, should we report a bug with LLVM?

Also I lost track here of one thing: the original issue @gnzlbg had was with inner bytes getting lost. I understand now that was due to ScalarPair. With that fixed, to clang/GCC guarantee to preserve inner padding bytes? Like, with a type such as

struct S { short x; int y; }
union U { struct S s; }

is the padding "between" s.x and s.y preserved? (Maybe your godbolt links answer that but I cannot interpret what that assembly is telling me.)

Because the violations @eddyb has in his test case are all about "trailing" padding, from what I can see. I could imagine specifying the value representations of something like Align16<u8> as all sequences of bytes of length 1. We'd have to find a good way to spec that, but it seems not entirely impossible, and it would still be a "bag of bytes" for all the bytes that are actually "covered" by any "field" (for some definition of "cover" and "field").

[comex]

Like, with a type such as

struct S { short x; int y; }
union U { struct S s; }

is the padding "between" s.x and s.y preserved? (Maybe your godbolt links answer that but I cannot interpret what that assembly is telling me.)

In that case it may be, but in the example I previously posted a Godbolt link for, it is not:

struct S {
    uint8_t a;
    uint64_t b;
};
union U {
    struct S s;
    uint8_t q;
};

In this case, Clang copies the first half of the struct using a 32-bit register move, so bytes 4 to 7 are lost. (Why 32-bit and not 8-bit? Likely because the resulting assembly is shorter.)

This particular ABI could theoretically be extended backwards-compatibly to preserve padding bytes. When a small integer argument or return value is passed in the lower bits of a (64-bit) register, whoever writes that register (i.e. the caller for arguments, callee for returns) is allowed to put anything it wants in that register's upper bits. So it could decide to preserve the full 64-bit register when making copies, since 'the original value' is as good as anything else to put in the upper bits. In theory Rust could guarantee that it does so, so that at least Rust-Rust calls would preserve the padding bytes (even when using extern "C"). Or the ABI specification itself could even be amended.

But not all ABIs work like that. The RISC-V ELF spec appears to require callers to zero- or sign-extend arguments in registers in a particular way. In other words, it requires the upper bits (which correspond to padding bytes) to have a specific value, and the callee can assume that they do have that value. The Apple AArch64 ABI (but not the non-Apple one) does the same. Under these ABIs, there is no (reasonable) backwards-compatible way to extend the C calling convention to preserve padding bytes, because there's nowhere to stash them.

[eddyb]

@RalfJung Turns out @gnzlbg was mistaken and no standard guarantees all bytes of a union are preserved, so we cannot guarantee much about #[repr(C)] unions, at least, when passed through extern "C" functions (even if they are both in Rust). None of this has anything to do with "scalar pairs" once we fix that, we're forced by ABIs to ignore some bytes of certain aggregates.

The fact that interior padding is preserved is just Rust using the whole register when GCC uses the bottom 32 bits and clang uses the bottom 8 bits - all 3 options are valid implementations of the SysV ABI (i.e. anything that's padding is effectively undef, so you can put whatever in them).

EDIT: oops, I missed @comex saying pretty much the same, but with more details.

[gnzlbg]

To summarize: repr(transparent) is correct in preserving or ignoring padding bits depending on the ABI of the type being wrapped.

I've opened https://github.com/rust-lang/unsafe-code-guidelines/issues/156 to track documenting this new bit of trivia for repr(C) unions.

I don't know how this information affects the guarantees that we could make about the layout of repr(Rust) unions, but it would be better for any discussion about that to continue in the UCG repo issue, which is currently: https://github.com/rust-lang/unsafe-code-guidelines/issues/73

[RalfJung]

I don't know how this information affects the guarantees that we could make about the layout of repr(Rust) unions

What remains on-topic here I think is the following observation: we cannot both support repr(transparent) and say that non-repr(C)-unions are bags of bits. Correct?

[gnzlbg]

What remains on-topic here I think is the following observation: we cannot both support repr(transparent) and say that non-repr(C)-unions are bags of bits. Correct?

Depends on whether we can make repr(C) unions bags of bits. These aren't bag of bits in C, but maybe there is a way to make them bag of bits in Rust ?

[mjbshaw]

I'm personally against the bag-of-bits idea. Some of my concerns/objects to this are that:

• Once you start dealing with the fields of a union, the padding bits are toast. Example:

  union U {
  field: (u8, u32),
  }
  
  fn set(pair: &mut (u8, u32)) {
  *pair = (1, 2);  // Padding bits are now garbage.
  }
  
  fn consume(_pair: (u8, u32)) {
  // _pair's padding bits are garbage.
  }
  
  pub fn do_stuff() {
  let mut u: U = unsafe { std::mem::zeroed() };
  set(unsafe { &mut u.field });
  consume(unsafe { u.field });
  }

  Neither set nor consume preserve the padding bits. The bag-of-bits idea might lead developers to incorrectly assume the padding bits are preserved in set or consume.

• Padding bytes should never be read (aside from doing stuff like memcpy). They're garbage. Preserving something's value is only useful if the value can be observed. The bag-of-bits idea implies it is okay to observe padding bytes, which I don't think we should do.

@RalfJung

It's the only definition I have seen so far that I actually know how to make precise, and that is not extremely complicated. I also think it is simple to teach, and I don't know any other definition that I would dare to even try teaching. Do you have a proposal for another one?

The idea I proposed earlier is, I think, both precise and simple:

• The value bits of a Rust union are the (mathematical) union of the value bits of its fields.
• The padding bits of a Rust union are the intersection of the padding bits of its fields.
• Passing/returning/reading a union (as a whole) preserves all value bits. Padding bits always have unspecified values and aren't preserved.

This definition is compatible with all reprs.

What remains on-topic here I think is the following observation: we cannot both support repr(transparent) and say that non-repr(C)-unions are bags of bits. Correct?

I think that's correct. You could still say that repr(Rust) unions are bags-of-bits (which I'm personally against).

@gnzlbg

Depends on whether we can make repr(C) unions bags of bits. These aren't bag of bits in C, but maybe there is a way to make them bag of bits in Rust ?

This can't be done reliably due to FFI and interactions with C. You might be able to get away with it as long as you never leave Rust-land, but I think drawing boundaries like that would end up with a picture that looks an awful lot like a foot-gun.

[gnzlbg]

Right now padding is only a function of field offsets. For unions to have padding, we need to incorporate that into our more general definition of layout somehow (eg also for structs).

Then there is also the issue of what happens when an union field is modified. Since we want to allow taking references to fields, we cannot forbid field padding from becoming garbage. If when accessing the union then this padding is “undef” as opposed to “frozen”, we have essentially introduced the notion of an “active field”/“field visibility”/“last modified field” in Rust, since depending on which field was modified last, some bytes of the union are undef, and we can optimize accordingly if we are able to track these modifications, which a good optimizer should then aim to track.

Rust does not have TBAA, so we don’t run into problems when creating two raw pointers to two different union fields, and we modify the union through both (eg in C++ both pointers can be assumed not to alias because they point to objects of different types).

So it would be worth it to try to figure out whether we can give all unions the same layout, including repr(C) ones, without introducing padding as a layout property and the notion of a last modified union field. If we have to introduce those, we have to balance the “optimization” vs “foot guns” spectrum.

For example, in practice, one cannot really implement an UB detector for C++ or C that tracks UB for unions (eg It is out of scope for UBSan, valgrind and all other existing tools to do this check). We do have as a constraint to be able to detect all UB with miri.

[RalfJung]

Neither set nor consume preserve the padding bits. The bag-of-bits idea might lead developers to incorrectly assume the padding bits are preserved in set or consume.

IMO that's a bad argument. If you change the union to

union U {
  field: (u8, u32),
  other: u64,
}

then all bits are preserved and your example still kills padding.

What we have to communicate clearly is that the type you use for the assignment matters. I call this a "typed copy" because it also occurs when passing arguments to a function or returning them back (where only the latter looks like an assignment in surface Rust).

Padding bytes should never be read (aside from doing stuff like memcpy). They're garbage. Preserving something's value is only useful if the value can be observed. The bag-of-bits idea implies it is okay to observe padding bytes, which I don't think we should do.

The hard part is defining what exactly "padding bytes" are in a union. That is a whole bag of complications that I had hoped to avoid.

If when accessing the union then this padding is “undef” as opposed to “frozen”, we have essentially introduced the notion of an “active field”/“field visibility”/“last modified field” in Rust, since depending on which field was modified last, some bytes of the union are undef, and we can optimize accordingly if we are able to track these modifications, which a good optimizer should then aim to track.

I do not think this is accurate. On every write, we perform a typed copy with the type used for that write. This is equivalent to transmuteing the value used for the write into a byte slice of appropriate length, and writing that.

No "active field" is needed to explain that behavior.

Miri currently does not even handle struct/enum padding on copies properly (i.e., it is preserved); see https://github.com/rust-lang/miri/issues/845. But once we find a good way to fix that, I think using that same mechanism also for unions should not be a problem.

[mjbshaw]

For example, in practice, one cannot really implement an UB detector for C++ or C that tracks UB for unions

I don't think that's possible without a tag, even for Rust.

The hard part is defining what exactly "padding bytes" are in a union. That is a whole bag of complications that I had hoped to avoid.

I think that's a prerequisite to having a productive conversation about this. If we're not on the same page and talking about the same thing then this whole conversation is doomed.

I'll go first. Here's what I have been meaning by the term "padding bytes": The padding bits of a Rust union are the union of the following:

• Any trailing padding bytes added to the end in order to align the type. These only exist when repr(align(N)) is used (and N is greater than the type's natural alignment).
• The intersection of the padding bits of its fields.

Is this the same or different from what you (and others) have meant by "padding bytes"? If it is the same, what do you think of my alternative to the bag-of-bits idea?

[RalfJung]

I think I'd like to start by defining the type of "padding bytes". My thinking is that it is a set of indices indicating the bytes of a type that are purely padding. So for example, for

#[repr(C)]
struct Test {
  f1: u8,
  f2: u16,
  f3: u8
  f4: u32,
  f5: u8
}

the "padding bytes" are: 1, 5, 6, 7, 13, 14, 15. Do we agree?

But then what are the "padding bytes" of an union enum?

This all makes me very sad because this notion just should not be needed. :(

[retep998]

This is how I interpret unions when working with them:

• Padding bytes in a value are always undefined. If I read a value from a union field, the padding bytes in the value are still undefined even if those bytes were well defined in the union through a write from a different variant. Reading that value does not change the status of whether a given byte is defined in a union.
• When writing a value to a union field, any bytes in the union that correspond to padding bytes in the value will be set to undefined. Even if they were earlier set to a well defined value through a different variant, they are now undefined.
• When copying a union wholesale, any byte that is padding for all variants of that union is considered padding for the union and its value is always undefined and does not have to be preserved during a copy. Any byte that was not well defined, either via not being written to in the first place or via having padding written to it, also does not have to be preserved during a copy and will remain undefined in the copy.

Is there any way that these rules could be stricter (in the sense of less code being legal)? If not, I'd like these to be the rules for what we guarantee for unions. We can always relax the rules later but I would be perfectly okay with this set of rules.