Proc macros for AsULE and AsVarULE

[Manishearth]

Part of https://github.com/unicode-org/icu4x/issues/1082

Depends on https://github.com/unicode-org/icu4x/issues/1078

It would be nice to be able to write:

#[derive(AsULE)]
struct Simple {
  a: u32,
  b: u32,
}

#[derive(AsVarULE)]
struct Relation {
   something: u8,
   something_else: u32,
   data: VarZeroVec<String>
}

and have it generate

#[repr(packed)]
struct SimpleULE {
   a: u32::ULE,
   b: u32::ULE
}

#[repr(packed)]
struct RelationULE {
   something: u8::ULE,
   something_else: u32::ULE,
   data: [u8]
}

[Manishearth]

For enums I can imagine us having derive(ULE) apply repr(u8) and just do transmutes. For larger enums this will fail and we'll have to have derive(AsULE) instead, but enums that large are not common anyway.

[Manishearth]

This will likely be a proc macro attribute, not a custom derive, with the proc macro attribute generating the ULE type. E.g. #[gen_var_ule(FooULE)] or something. Writing the ULE type correctly can be tricky so we probably need the proc macro to generate that part too, not just the ULE implementation.

Splitting this into different tasks it may have to do:

• ULE:
  • for structs: generate a repr(packed) struct containing ULE fields
  • for dataless enums: generate a PlainOldULE<n> wrapper with validation, perhaps requiring numeric values to be prespecified so that people don't accidentally break stability by reordering
  • for dataful enums: Do some kind of bit-packing. This might be best left up to manual impls
• VarULE:
  • for structs with a single DST-ish field: stick it at the end, use ULE versions of everything. This will be pleasant because you can still directly read from the fields of the ULE type if necessary.
    • We can start off with a hardcoded list of VarULE types but we can technically support marking multiple EncodeAsVarULE types as such with an attribute and providing their VarULE version
  • for structs with multiple DST-ish fields: stick a [u8] at the end and additional private info about the indices for the individual DSTs contained. Unlike the single-dst case we won't be able to allow for easy access to the field directly, but we can add methods, and as usual folks can use the FromVarULE impl
  • For enums with DST fields: we can technically do complex bit packing and have users rely on FromVarULE, but this also seems best left to manual impls.

For ULE types it would generate:

• The ULE type
• ULE and AsULE implementations
• a ZeroMapKV implementation

For VarULE types it would generate:

• The VarULE type
• the VarULE implementation
• EncodeAsVarULE and FromVarULE implementations
• Potentially getters for fields in the multiple-DST case

[Manishearth]

We may also need #[derive(ULE)] that allows wrapping ULE types with more ULE types, in case you wish to implement a custom conversion. Ideally you can also #[generate_serde(target)] on it to generate appropriate ser/de impls.

[Manishearth]

As for bit packing and enums, I think the way to do this is:

• A BITS constant on ULE that specifies the number of bits actually needed by the type
• By default the custom derive just generates a repr(packed) struct of ULEs.
• If you specify #[zerovec::bits(4)] on a field, the custom derive will instead:
  • Try to pack it in with any adjacent zerovec::bits fields
  • Write code const-asserting that BITS matches the given bitsize
  • Use struct BitPacker([u8; N]) to store the bits field, which has unsafe fn get_ule<U: ULE>(bit: usize) getting a ULE U at offset bit
  • Autogenerate getters for each field that got bitpacked

Dataful enums can then be done by mandating that all fields are marked with zerovec::bits and doing their own discriminantful bitpacking. VarULE for these will involve generating a more complex type, but it's still doable.

[Manishearth]

The design can be found in the design doc: https://github.com/unicode-org/icu4x/blob/main/utils/zerovec/design_doc.md#proc-macros