Open cinnamon-bun opened 4 years ago
In the meantime: Vanilla clients can import earthstar-graphql
and call syncGraphql
to sync with GraphQL pubs.
https://github.com/earthstar-project/earthstar-graphql/blob/master/src/sync-graphql.ts
I was thinking a lot about this. Your idea to add REST endpoints to the existing GraphQL server is pragmatic and a good stepping stone. But I feel like it doesn't address the underlying issue, which is — very understandably — that we don't want to have multiple kinds of pubs, and hence multiple kinds of syncing.
It would mean:
That's not what I want to contribute to this project.
I started building earthstar-graphql
from the perspective of a client author who would use it to build complex apps very quickly using a rich existing ecosystem of tools.
But after having built some things that talk to each other, I now see that a huge part of earthstar-graphql
's value is in being able to query and mutate earthstar peers using a common query language over any transport: HTTP, websockets, function calls, messages delivered by pigeon.
This is why I think it could be a good solution for RPC.
Because earthstar-graphql
is supposed to be able completely power an earthstar client, this means you can do a lot remotely: getting authors, workspaces, documents, triggering syncing, setting data to workspaces, etc.
And this made adding syncing to earthstar-graphql
very easy: you query another earthstar-graphql instance's documents the same way you would locally, and then ingest the resulting JSON into a IStorage
.
It also means there are possibilities for things like proxy stores, which @cinnamon-bun saw the potential for pretty much immediately, and made this diagram
What I'm proposing is adopting GraphQL as the way to communicate with pubs. Most importantly, I think we can do this in a way where no-one needs to know anything about GraphQL.
The core earthstar
package already does this with REST endpoints, where the implementation of syncing is abstracted behind syncLocalAndHttp
. Users don't need to know what the routes and methods are for fetching and mutating resources, they just call a function and earthstar
does it for them.
The same thing could be done with GraphQL, where users simply give an IStorage and pub URL, and the sync happens without anyone needing to know how, just like syncGraphql
does.
It wouldn't add any dependencies. All we need to do is send a query string over HTTP.
What we get from it is being able to easily add new capabilities to syncing (and I think Earthstar's syncing is going to be powerful!) without having to maintain our own parallel vocabulary of REST endpoints and all the possible params that can be given to them, because we'll have a powerful way of doing that at our disposal already.
There are other benefits, e.g. because GraphQL can support streaming updates, we can do things like continuous real-time syncing.
This is asking a lot, and I can completely understand if you'd say no to it. Earthstar wisely wants to use boring tech, and REST is more boring that GraphQL. But I feel like GraphQL's no spring turkey either. It's been five years since being released publicly and has a lot of documentation, tooling, and adopters.
People wanting to write their own earthstar libraries from scratch would need to learn a little about GraphQL, but the same is true of learning all the REST endpoints. But they could also copy and paste the GraphQL queries used by earthstar
in their own projects.
Otherwise, I think we'd be best off having one kind of pub, and one kind of syncing — and I think it'd be best for Earthstar if I removed the notion of doing these over GraphQL.
Problem
Vanilla Earthstar apps (just using the
earthstar
package) can't sync withearthstar-graphql-pub
pubs.Adding GraphQL to an app is not always easy. It adds dependencies and complexity and changes the internal API used for doing Earthstar stuff.
Proposal
Could GraphQL pubs also serve the basic HTTP endpoints so any earthstar client could sync with them?
One way to do this:
There would also be some trickery to make sure the HTTP pub and GraphQL pub code is using the same underlying IStorage objects.
earthstar-pub
's new exported code would be designed with that in mind, somehow.Quick research about this