Closed sgwilym closed 3 years ago
Awesome!
Should we call these sync queries or sync filters?
There are 2 kinds of them, incoming and outgoing. I haven't implemented them anywhere yet though.
Peer1 --> Peer1's outgoing filter ---- - - - ----> Peer2's incoming filter --> Peer2
Peer1 <-- Peer1's incoming filter <--- - - - ----- Peer2's outgoing filter <-- Peer2
The outgoing filter is what you are willing to share. Maybe you only want to upload /wiki/*
but not /photos/*
with a certain pub.
The incoming filter is what you want.
Documents have to pass through both filters to be successfully sync'd (they have to be offererd and wanted).
More details in this issue, including that you can stack multiple queries to combine them.
Also, eventually, for efficiency: Peer1 can ask Peer2 for its incoming filter and apply it before transmitting, to reduce the amount of transmitted data.
Peer2 still needs to apply it again, because it doesn't trust Peer1 to do it right.
one-way sync example. double this for 2-way sync
PEER1 network PEER2
=========================================== ========================
Peer1 --> Peer1.outgoing --> Peer2.incoming --- - - - - -----> Peer2.incoming --> Peer2
Thanks to your comments and fresh eyes this morning, I came back at this with quite a few changes.
Here’s what syncing looks like between two GraphQL peers with these new additions:
ingestDocuments
mutationSome things I noticed while implementing this:
Document
, particularly versionsByAuthors
, so I punted on it.What if a workspace’s sync filters were stored somewhere in its IStorage
? Then filtering could happen during document ingestion, and it would be easier to keep track of which workspace has which sync filters.
—
I also changed the sync
mutation’s name to syncWithPub
to make it clearer that this mutation would cause earthstar-graphql to initiate a sync rather than be on the receiving end of one (in which case it would receive an ingestDocuments
mutation).
This conceptual awkwardness is a side-effect of this lib being able to power a single client and a pub. 😬 I don’t think this’ll be the last time, either.
—
The two different context creation functions have been replaced by a single one. I think the schema context shape will become easier to manage if IStorage
is ever made to store multiple workspaces.
This adds a new exported function for syncing documents to a
IStorage
from a GraphQL endpoint, which means that earthstar-graphql servers can now act as pubs!As well as this, the
syncGraphql
function also supports filters, so peers can askearthstar-graphql
pubs for only a subset of documents they're interested in, i.e. efficient sync / partial replication.This new way of syncing has also been added as an argument to the
sync
mutation:Closes #5, Closes #6