Hyperlog-style semantics with hypercore/hyperdb?

[saranrapjs]

Not sure if this is the appropriate forum for this type of question, but: is there a straightforward way of achieving append-only ("hyperlog"-style) multi-writer semantics with hyperdb?

I like the sparse replicating aspects of hypercore/hyperdb, and hyperdb seems to be the only hypercore implementation which supports multiple writers, but I'm basically just interested in appending to a log (e.g. I don't need the overhead of the key/value store hyperdb provides). I noted multifeed, which sort of seems to hint something in this direction?

At any rate, happy to be told that hyperdb is not the tool for this particular architecture, but would be curious to hear if there are plans/designs for something like what I'm describing! And if there's a better place to ask these kinds of questions :)

[hackergrrl]

I'd really like this too. Having a multifeed module would be a good base for making something that can implement hyperlog's API. Steps (in my mind):

1. Finish multifeed (multicore?) (this could be just forking hyperdb and stripping the kv parts out)
2. Write an implementation of the hyperlog API on top of that (corelog? hyperfeed?)

[pfrazee]

Would it be sufficient to use hyperdb with a keyspace of monotonic integers? That is, you could just write to /1, /2, /3, etc.

[saranrapjs]

(this could be just forking hyperdb and stripping the kv parts out)

I hadn't thought about this! Might give this a try, time permitting.

Would it be sufficient to use hyperdb with a keyspace of monotonic integers?

I thought about this too...I haven't closely inspected how the various hypercores in hyperdb get reconciled, but I'm assuming that the topographic ordering referred to in the README could lead to situations where a peer would inadvertently overwrite another peer's key.

Though, something that's actually just now occurring to me: it might be possible to use db.createHistoryStream as the append-only log, and have all peers just consistently write to the same key? You'd essentially use the topographic ordering to get at monotonicity...possibly at the cost of all of the overhead used to materialize the key/value parts of hyperdb?

[pfrazee]

I'm assuming that the topographic ordering referred to in the README could lead to situations where a peer would inadvertently overwrite another peer's key.

Yeah that would happen. I guess the solution depends on how you want your log to behave. Internally hyperdb is a set of logs which get woven together. If you use the history stream by itself, I'd expect that you're going to get different orders for each device, but certainly no conflicts.

[hackergrrl]

You could also go under the covers and access db._writers directly, which is an array of Writers. Each Writer has a .feed field that's that writer's hypercore instance. Then you could just ignore the regular hyperdb kv API and manage the hypercore feeds directly. Though a proper multifeed module is a much more reusable way to go I think.

[mafintosh]

noting that i have multifeed on npm in case anyone wants to take a stab at that. originally i wanted to build something on top of a multifeed module but in practice that is really tricky as you really need the trie internally to make this thing scale if you want random access replication

[mafintosh]

what @pfrazee is suggesting is actually a really good solution btw as it gives you an implicit notion of time. i.e. if a fork happened a loooooooong time ago it's "seq" will be /{small-number} vs the latest changes being /{large-number}.