search

hldb / welo

peer-to-peer, collaborative states using Merkle-CRDTs
Other
32 stars 2 forks source link

feat: bootstrap and pubsub replicators #87

Closed saul-jb closed 11 months ago

saul-jb commented 1 year ago

This PR adds two new replicators: pubsub and bootstrap.

The pubsub replicator is a simplified version of the live replicator, sending heads on a shared pubsub channel. The bootstrap replicator uses content routing to locate peers with the database block then requests heads from them.

These two replicators work in conjunction with each other, the pubsub replicator can only handle live updates whereas the bootstrap replicator is for obtaining heads from peers when it first starts.

Note that the bootstrap replicator relies on a DHT service to be added to the Libp2p config.

Both of these currently lack tests and may contain bugs.

saul-jb commented 1 year ago

~The bootstrap replicator seems to not work if the in-memory blockstore is used but works fine with a FsBlockstore, not sure why... needs more investigation.~

The bootstrap replicator gets stuck trying to fetch a block regardless of what blockstore is used.

tabcat commented 1 year ago

These looks really great! The pubsub replicator is a much over the live replicator. The bootstrap replicator module finds peers of the database by querying the dht for providers of the database manifest. Can't wait to make these default and replace the live replicator.

tabcat commented 11 months ago

Going to merge this since all but chrome tests are passing and people have been wanting to use it.

These changes affected more than I anticipated so not going to merge this yet. Will get https://github.com/hldb/welo/pull/103 merged first. Large parts of this may need to be rewritten, mostly tests, but will make sure it's easy.