hsanjuan
commented
5 months ago Notes from going over:
instances can use metadata or summary vectors of their snapshots. These summaries allow instances to quickly ascertain if they need the full snapshot or if they're already up-to-date: I don't see how the state is represented or how "summaries" are represented. I don't think you can have a syncing mechanism that involves exchanging full snapshots. That doesn't scale with huge indexes nor indexes that are updated very frequently by several parties. There needs to be a notion of "delta", and how the state is represented is important to have a way to create those deltas. ie. perhaps representing the state as a merkle-dag can facilitate diffing and snapshot sharing: https://ieeexplore.ieee.org/document/9049566 (a perhaps better alternative to merkle-crdts from go-ds-crdt mentioned above).- Broadcast mechanisms are not detailed: how does a broadcast here work? Careless broadcast will become the first bottleneck on a federated system. Is that an HTTP POST endpoint? Is that other protocol?
- How are snapshots authenticated? Is the CID a hash of the full snapshot? What happens when snapshots are huge?
Specify the initial IPNI federation protocol which aims to achieve eventually consistent index records across a collaborating set of nodes.
The federation protocol consists of four fundamental steps: Initialization, Periodic snapshot taking, Exchange of snapshots and Reconciliation. The protocol takes advantage of the immutability of advertisements exposed by each provider to resolve conflicts across indexers.
The specification lists a set of APIs exposed by a participating indexer in order to enable the implementation of the federation protocol.
See rendered document.