After a few experiments, we can introduce delta-based quicksync.
From the client side, the process of quicksync would look like:
IF the DB exists, just download batches and restore batch after batch
IF DB does not exist, download normal quicksync (full db state)
Process of batch downloading:
each batch is produced with start_layer,end_layer,start_layer_minus_one_db_hash. The layer hash NEEDS TO BE COMPARED before restoring with the local state
IF hash is different then proceed to PREVIOUS batch till the hash matches
after restoring from partial you can continue without layer checks.
The server side publishes partials in predefined sizes OR times so do not hardcode the start-end span. All values in metadata.csv are sorted by start_layer.
Restoring same partial twice should be noop and is considered safe. BE WARNED that there are NO deletes during the restore process (that's why absolutely make sure that you compare the layer_hash).
It is safe to stop / restart / retry at any stage after restoring from any number of partials
Objects that are not tied to time or layer are remembered on the server side and added to the first batch when they appeared in the DB.
After a few experiments, we can introduce delta-based quicksync.
From the client side, the process of quicksync would look like:
Process of batch downloading:
start_layer,end_layer,start_layer_minus_one_db_hash. The layer hash NEEDS TO BE COMPARED before restoring with the local stateThe server side publishes partials in predefined sizes OR times so do not hardcode the start-end span. All values in
metadata.csvare sorted by start_layer.Objects that are not tied to time or layer are remembered on the server side and added to the first batch when they appeared in the DB.
The process should be integrated into
quicksync-rsand the initial integration was provided in https://github.com/spacemeshos/quicksync-rs/pull/49