earthstar-project / earthstar

Storage for private, distributed, offline-first applications.
https://earthstar-project.org
GNU Lesser General Public License v3.0
624 stars 18 forks source link

Persist document syncing thumbnails #290

Open sgwilym opened 1 year ago

sgwilym commented 1 year ago

What's the problem you want solved?

In v10 Earthstar adopted range-based set reconciliation to make sync more efficient.

One of the steps of this methodology involves creating a large augmented red black tree which is able to produce 'fingerprints' for certain ranges of items.

Currently, this tree is created from scratch every time a replica is synced for the first time.

If a replica held a large number of documents, redoing this work every time could impact the experience of syncing.

It would be good to be able to persist these fingerprint trees between sessions in order to avoid redoing this work.

Is there a solution you'd like to recommend?

I'm not sure yet. Whatever method we'd choose would have to be careful not to let the documents on the replica and the document thumbnails on the fingerprint tree get out of sync.