ArcCastPtr / BoxCastPtr: allow `?Sized` types

In https://github.com/rustls/rustls-ffi/pull/341#issuecomment-1745711553 we talked about representing a Rust Arc<dyn ClientCertVerifier> in rustls-ffi. Our utility traits like ArcCastPtr represent a smart pointer (like Arc), and do casting between C pointers and Rust *const raw pointers, where the pointee type is the inner type (in this case dyn ClientCertVerifier).

The first stumbling block we run into is the implicit Sized bound on the RustType associated type:

pub(crate) trait CastConstPtr {
    type RustType; // this is Sized by default

    fn cast_const_ptr(ptr: *const Self) -> *const Self::RustType {
        ptr as *const _
    }
}

However, when we explicitly disavow the Sized bound (with RustType: ?Sized), we run into other problems:

error[E0606]: casting `*const Self` as `*const <Self as CastConstPtr>::RustType` is invalid
 --> foo.rs:7:9
  |
7 |         ptr as *const _
  |         ^^^^^^^^^^^^^^^
  |
  = note: vtable kinds may not match

If we also try to remove the + Sized bound on ArcCastPtr, we get another, more instructive error:

trait CastConstPtr {
    type RustType;

    fn cast_const_ptr(ptr: *const Self) -> *const Self::RustType {
        ptr as *const _
    }
}

trait ArcCastPtr: CastConstPtr {
    fn to_const_ptr(src: Self::RustType) -> *const Self {
        Arc::into_raw(Arc::new(src)) as *const Self
    }
}

error[E0607]: cannot cast thin pointer `*const <Self as CastConstPtr>::RustType` to fat pointer `*const Self`
  --> foo3.rs:13:9
   |
13 |         Arc::into_raw(Arc::new(src)) as *const Self
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Some references:

https://doc.rust-lang.org/stable/reference/types/trait-object.html https://doc.rust-lang.org/stable/reference/dynamically-sized-types.html https://geo-ant.github.io/blog/2023/rust-dyn-trait-objects-fat-pointers/

That blog post is perhaps the most straightforwardly informative:

if you’re like me you will be (or already were) surprised to learn that the pointer types Box, &T, and &mut T are different from the pointer types Box, &dyn Trait, and &mut dyn Trait. The latter are fat pointers. They, again, consist of two elements: their first element is the pointer to the actual data (the T instance) and the second is the pointer to the associated vtable instance (the Trait-vtable for type T).

So I think what's going on here is that trait ArcCastPtr: CastConstPtr + Sized means Self is sized, which means it can be pointed to with a thin pointer. But when we remove the Sized bound, suddenly Self could be a dynamically-sized type (DST), which means *const Self could be a fat pointer (that is, a pointer to a dyn T or a [T]). Casting between thin pointers and fat pointers is not allowed, because information could be lost!

I'm not sure at the moment what the right solution is. In that PR we came to a simple solution by wrapping the Arc in an outer Box, which is fine, but I wanted to understand the underlying problem. Leaving this issue here to document what I've found so far.

[Edit: removed some stuff about trait object safety which on further reflection I concluded was wrong.]

rustls / rustls-ffi

ArcCastPtr / BoxCastPtr: allow `?Sized` types #350