jean-pierreBoth / hnswlib-rs

Rust implementation of the HNSW algorithm (Malkov-Yashunin)
Other
157 stars 22 forks source link

Change borrow to move semantics for HnswIo index loading #19

Open bwsw opened 4 months ago

bwsw commented 4 months ago

Hello, now when you try to load the database from the disk, the code borrows structures from HnswIo. It produces a very uncomfortable use condition, either requiring a singleton or, as I recently did - a thread or async with a loop, so the borrow checker can ensure the correctness. Both variants are very constraining.

I propose changing this behavior to "move" behavior when Hnsw consumes HnswIo rather than borrows its properties. As a result, its use will be significantly simplified. However, I still do not understand the role of this borrowing design because, from my perspective, HnswIo is just a loader, not a holder of loaded data.

If particular data must be used by the Hnsw index, it needs to be moved there. We can use various approaches like:


enum Data {
   DataWithoutSideEffect(bla),
   DataWithSideEffect(blabla)
}
jean-pierreBoth commented 6 days ago

Yes the borrow is cumbersome, i battled with that. The problem is Hnswio keeps mmap data and allows a partial memory load to access a point without loading the structure.

I did not find a better solution or give the Datamap responsability to the user.