turbofish-org / merk

High-performance Merkle key/value store
Apache License 2.0
226 stars 36 forks source link

Aux data API #30

Closed mappum closed 4 years ago

mappum commented 4 years ago

We'll need to store some auxiliary data in the store such as the most recent blockchain height or version information, which doesn't need to be part of the Merkle tree but will still need to be part of the same atomic commit. This should probably be stored in a separate column family.

We can possibly create a method like apply_aux(tree_ops: Batch, aux_ops: Batch).

davebryson commented 4 years ago

As mentioned in the earlier issue related to 'polish API', I think this signature on Merk would make that a bit easier, as the application would have access to the Database in the abci.commit() to store the chain info (in another column) vs the Database handle being embedded in Merk.

pub fn new(db: Arc<dyn Database>) -> Result<Merk> {}
mappum commented 4 years ago

Unfortunately, that wouldn't make it possible to atomically commit the auxiliary values at the same time as the changes to the tree.

If we want to expose the db, we should probably just add a fn database(&mut self) -> &mut Merk method, since then it doesn't need an Arc and a trait object.

Also, I'm still opposed to a Database trait since we expect to depend on so many RocksDB-specific features, e.g. snapshots, OptimisticTransaction, certain tuning options, etc. If we wanted to unbind from RocksDB and make this generic, the trait would essentially end up just being the same as the RocksDB API and be very difficult to implement for anything that isn't RocksDB.

mappum commented 4 years ago

I realize this change belongs in Merk::commit rather than apply, so maybe we should just add an aux: &Batch argument there.

While we're at it, we should move the special key which points to the key of the root node (https://github.com/nomic-io/merk/blob/develop/src/merk/mod.rs#L250-L254) to the auxiliary column family (or even a third column family) so it can't conflict with the core tree keys.