tursodatabase / libsql

libSQL is a fork of SQLite that is both Open Source, and Open Contributions.
https://turso.tech/libsql
MIT License
9.54k stars 252 forks source link

replicate from stored segments #1608

Closed MarinPostma closed 2 months ago

MarinPostma commented 2 months ago

This PR makes the necessary changes to perform replication from already stored segments.

Replication from a stored segment is performed when the requested frame number is not in the current log or the sealed segment list. We may then need to fetch those frames from durable storage.

After replicating all the frames we could from the current and sealed logs, we fetched stored segment indexes. If the index contains a frame we haven't seen already, we fetch the segment and replicate missing frames. We do that until we have replicated all the pages required until the requested frame_no.

As an optimization, we need to implement caching of segments and indexes to avoid fetching segments over and over again.