mafintosh / hyperdb

Distributed scalable database
MIT License
753 stars 75 forks source link

How to efficiently get all heads *prior* to a node? #48

Open hackergrrl opened 6 years ago

hackergrrl commented 6 years ago

When building a hyperdb-index, a common desire is to, when processing a node, to remove all previous heads from the index for that key, and replace them with the new value.

With hyperlog, node.links gave those hashes. How could this be done with hyperdb? I hacked it together in a roundabout way:

  1. modify the new node's node.clock back one entry for its local feed
  2. convert node.clock into a buffer of heads compatible with hyperdb#version
  3. do a hyperdb#checkout of that version (one entry before this new node was inserted)
  4. do a lookup on node.key: these are the old heads

This works, but it's a pretty heavy process, especially in the tight loop of an indexer.

@mafintosh can this be done more efficiency with the already existing incremental index?