The idea for the VTables change is that it is encodings who implement the behavior of arrays taking and returning type-erased ArrayDatas.
This was already the case for canonicalize, this PR just formalizes it as a separate behavior.
It is also a useful function to call directly on arrays, hence why the non-vtable trait IntoCanonical still exists and is used to derive the IntoCanonicalVTable impls.
I still need to play around with non-owned vtable functions, the restriction being E::Array: TryFrom<ArrayData> would need to become for<'a> &'a E::Array: TryFrom<&'a ArrayData>, which is obviously impossible.... unless we only implement it for arrays that are #[repr(transparent] to an ArrayData. In other words, the macro'd BoolArray struct would need to lose its metadata field and become a unit struct that only holds ArrayData.
The implication here is that either:
Metadata needs to be parsed on every access, and OwnedArrayData simply holds a Buffer for metadata.
We keep ArrayData holding an Arc<dyn ArrayMetadata> and downcast it on access, but we still need to parse metadata for ViewedArrayData. Although I suppose we could also parse that on construction into Arc<dyn ArrayMetadata>....
FLUPS:
[x] Experiment with whether the heap allocation to parse metadata for ViewedArrayData is too costly. If so, we should make all metadata constant time access from &[u8], i.e. move to flat buffers or C-repr metadata. If not, we can hold Arc<dyn ArrayMetadata> on both Owned and Viewed ArrayData, meaning we can remove it from the SomeArray struct and therefore permit the transmute from &ArrayData to &SomeArray.
Therefore for now I suggest we deserialize metadata on construction of ArrayData, even for ViewedArrayData. This caches the deserialized metadata for the lifetime of the ArrayData, instead of deserializing on every access.
In the future, we could make this Option<Arc<dyn ArrayMetadata>> to allow an array to decide whether to eagerly load metadata or do it lazily.
gatesn
commented
5 days ago
The idea for the VTables change is that it is encodings who implement the behavior of arrays taking and returning type-erased ArrayDatas.
This was already the case for canonicalize, this PR just formalizes it as a separate behavior. It is also a useful function to call directly on arrays, hence why the non-vtable trait IntoCanonical still exists and is used to derive the IntoCanonicalVTable impls.
I still need to play around with non-owned vtable functions, the restriction being
E::Array: TryFrom<ArrayData>would need to becomefor<'a> &'a E::Array: TryFrom<&'a ArrayData>, which is obviously impossible.... unless we only implement it for arrays that are#[repr(transparent]to an ArrayData. In other words, the macro'dBoolArraystruct would need to lose its metadata field and become a unit struct that only holds ArrayData.The implication here is that either:
Arc<dyn ArrayMetadata>and downcast it on access, but we still need to parse metadata for ViewedArrayData. Although I suppose we could also parse that on construction intoArc<dyn ArrayMetadata>....FLUPS:
&[u8], i.e. move to flat buffers or C-repr metadata. If not, we can holdArc<dyn ArrayMetadata>on both Owned and Viewed ArrayData, meaning we can remove it from the SomeArray struct and therefore permit the transmute from&ArrayDatato&SomeArray.Option<Arc<dyn ArrayMetadata>>to allow an array to decide whether to eagerly load metadata or do it lazily.