meilisearch / heed

A fully typed LMDB wrapper with minimum overhead 🐦
https://docs.rs/heed
MIT License
632 stars 57 forks source link

Combined version of LMDB mdb.master and mdb.master3 #278

Open Kerollmops opened 3 months ago

Kerollmops commented 3 months ago

This PR implements #51, taking a large inspiration from the im-rs crate .

It uses a second heed3/Cargo.toml with the necessary dependencies. When you need to work or publish the heed3 crate, you need to cp heed3/Cargo.toml heed/Cargo.toml. The examples were moved out of the standard heed/examples/ folder to make them compile when working on both crates.

There are three crates now...

...but I want to only have two

Note that if we duplicate the code of the Database for the EncryptedDatabase type, we no longer need the following proc-macro as we will copy/paste and change the method signature, i.e., &RoTxn -> &mut RoTxn, by hand.

How it works?

By annotating all the heed methods that use a &RoTxn this way:

#[heed_master3_proc_macro::mut_read_txn(rtxn)]
fn get<'t>(&self, rtxn: &'t RoTxn, key: &[u8]) -> heed::Result<&'t [u8]>;

It transforms the function signature to use a &mut RoTxn:

fn get<'t>(&self, rtxn: &'t mut RoTxn, key: &[u8]) -> heed::Result<&'t [u8]>;

This way, we ensure that users do not keep pointers for potentially invalid bytes from LMDB between two get/put operations. It's a limitation of LMDB when you use the encryption feature. The LMDB pages are decrypted on the fly in a buffer that cycles. As a result, only a restricted amount of values' pointers are valid until the next operations.

To Do