meilisearch / heed

A fully typed LMDB wrapper with minimum overhead 🐦
https://docs.rs/heed
MIT License
519 stars 52 forks source link

Make `ReservedSpace` more flexible #254

Closed nolanderc closed 2 months ago

nolanderc commented 3 months ago

The only way to use ReservedSpace currently is to use the std::io::Write implementation. However, this assumes that the entire value can be written in one go. This prevents certain patterns, such as having a header in the value which contains the a checksum of the remaining bytes.

There should be some mechanism to reference arbitrary bytes within the space. Some options (naming TBD):

  1. Add a method to get all previously written bytes: fn written_bytes_mut(&mut self) -> &mut [u8]. This is safe as the bytes are known to be initialized.
  2. Add a method to zero-initialize and get a mutable view of the entire space: fn zero_bytes_mut(&mut self) -> &mut [u8]. This would mark the entire space as written.
  3. Add a method fn as_bytes_mut_uninit(&mut self) -> &mut [MaybeUninit<u8>]. Can be used to write arbitrary data in any order. Needs a complementary method to mark them as written: unsafe fn assume_written(&mut self, count: usize), which is unsafe because caller must guarantee that all bytes 0..count have been initialized.
Kerollmops commented 3 months ago

Thank you for this issue. It also summarizes some of my previous ideas. I wanted to only expose a &mut [MaybeUninit<u8>] to users and let them write to it using the MaybeUninit::write_slice method. Unfortunately, it's still unstable...