xacrimon / dashmap

Blazing fast concurrent HashMap for Rust.
MIT License
3.09k stars 151 forks source link

Pre-checks before writing a backup/persistence layer for DashMap #241

Open laralove143 opened 1 year ago

laralove143 commented 1 year ago

My idea is simple, the crate would dump/update the data to a file using its serde feature in another thread periodically or for every write

The use-case would be when you want to combine sled and DashMap but want to put memory first or memory usage isn't a concern

Is this possible or a good idea? If this gets an approve I'll start writing it

xacrimon commented 1 year ago

This is interesting but I'm sure exactly how this would work. This seems to be like you essentially want to run a full in-memory cache in front of sled? If that's what you're looking to do, it's probably easiest just to create a wrapper type over dashmap that inserts -> writes to sled -> returns.

laralove143 commented 1 year ago

Using sled is probably the best idea, but sled works with [u8] while DashMap can work with Serialize, do you know a sled-alternative that works with serde? Of course you could convert a serialized value to a [u8] but I don't think that's ideal. It would be even better if the library could support a cost-optimized serialization, do you know a library like that? Or if I'd use sled do you know a cost-optimized adhoc serialization library?