meilisearch / heed

A fully typed LMDB wrapper with minimum overhead 🐦
https://docs.rs/heed
MIT License
569 stars 52 forks source link

Implement Send for RoTxn #191

Closed MarinPostma closed 1 year ago

MarinPostma commented 1 year ago

I wanted to make the write transactions Send but LMDB's documentation says:

A write Transaction may only be used from the thread it was created on.

However, I think that making RoTxn: Sync with the sync-read-txn is not safe, but I may be mistaken. But, for sure, RoTxn should implement Send, since it can be moved to other threads safely.

Kerollmops commented 1 year ago

According to the documentation it seems that you are right we can't refer to the read transaction from different threads, at least we must synchronize the use (by using a mutex?). According to Howard Chu, we must remove this sync-read-txn feature and introduce the new send-read-txn feature you want to implement instead.

What's cool is that if we implement RoTxn: Send, users will be able to use a Mutex to use it through threads as it only requires T: Send for the Mutex<T>: Sync.

MDB_NOTLS

Don't use Thread-Local Storage. Tie reader locktable slots to #MDB_txn objects instead of to threads. I.e. #mdb_txn_reset() keeps the slot reserved for the #MDB_txn object. A thread may use parallel read-only transactions. A read-only transaction may span threads if the user synchronizes its use. Applications that multiplex many user threads over individual OS threads need this option. Such an application must also serialize the write transactions in an OS thread, since LMDB's write locking is unaware of the user threads.

Yanking or Reporting a Security Issue?

I have some questions for the crates.io team or Rust developers in general, but should I yank all previous versions of the heed crate? Or should I raise a RustSec advisory about heed that concerns this specific sync-read-txn feature? The latter seems more appropriate as the library is safe to use until you use this particular feature.

Thank you for the investigation and report again 🙏

Kerollmops commented 1 year ago

Reopening as I need to fill a RustSec advisory.

Kerollmops commented 1 year ago

I just asked for help in https://github.com/rustsec/advisory-db/issues/1755, and we will continue the discussion there, closing.