Currently, the sync implementation is based on syncing individual items. It retrieves an item (ID) list from two servers, compares them, and copies across any missing items.
However, the time to do that to remote servers is entirely dominated by round-trip latency. Even with a parallelism of 4, we can only reduce that O(n) to ... a slightly smaller O(n).
Instead, we could add a way to get items IDs and their contents, streamed.
The client can just sift through these and discard the ones it doesn't want, or close the stream when it gets to a section of items it already has.
Optionally: Add an option to stop the stream (server-side) at a particular timestamp or signature.
Currently, the sync implementation is based on syncing individual items. It retrieves an item (ID) list from two servers, compares them, and copies across any missing items.
However, the time to do that to remote servers is entirely dominated by round-trip latency. Even with a parallelism of 4, we can only reduce that O(n) to ... a slightly smaller O(n).
Instead, we could add a way to get items IDs and their contents, streamed.
The client can just sift through these and discard the ones it doesn't want, or close the stream when it gets to a section of items it already has.
Optionally: Add an option to stop the stream (server-side) at a particular timestamp or signature.