search

turbofish-org / merk

High-performance Merkle key/value store
Apache License 2.0
226 stars 36 forks source link

Concurrent operations #26

Open mappum opened 4 years ago

mappum commented 4 years ago

Everything other than the RocksDB compaction is currently run in the same thread. We should be able to trivially split up the first pass (changing the tree structure) across N workers with no synchronization other than joining left and right. Splitting up the second pass (hashing and writing to RocksDB) can be done similarly, but will either require an efficient data structure for merging the different batches of RocksDB writes, or even possibly writing to RocksDB in separate workers with some kind of protocol to make the separate writes atomic (e.g. each is written with an OptimisticTransaction, with all of them committed in a single commit operation if possible).

MuhtasimTanmoy commented 10 months ago

Is this requirement active?