compatability with HTTP pubs?

[cinnamon-bun]

Problem

Vanilla Earthstar apps (just using the earthstar package) can't sync with earthstar-graphql-pub pubs.

Adding GraphQL to an app is not always easy. It adds dependencies and complexity and changes the internal API used for doing Earthstar stuff.

Proposal

Could GraphQL pubs also serve the basic HTTP endpoints so any earthstar client could sync with them?

One way to do this:

• Change earthstar-pub so it exports a reusable express app object with the HTTP routes
• Import that into earthstar-graphql-pub and run it alongside the Apollo server, so the Apollo server also has the HTTP pub routes

There would also be some trickery to make sure the HTTP pub and GraphQL pub code is using the same underlying IStorage objects. earthstar-pub's new exported code would be designed with that in mind, somehow.

Quick research about this

• https://www.apollographql.com/docs/apollo-server/v1/servers/express/
• https://spectrum.chat/apollo/apollo-server/using-express-alongside-apollo-server~0da4a3f3-e441-46b8-987e-de3374fb3e66

[cinnamon-bun]

In the meantime: Vanilla clients can import earthstar-graphql and call syncGraphql to sync with GraphQL pubs.

https://github.com/earthstar-project/earthstar-graphql/blob/master/src/sync-graphql.ts

[sgwilym]

I was thinking a lot about this. Your idea to add REST endpoints to the existing GraphQL server is pragmatic and a good stepping stone. But I feel like it doesn't address the underlying issue, which is — very understandably — that we don't want to have multiple kinds of pubs, and hence multiple kinds of syncing.

It would mean:

• More maintenance work for earthstar contributors (no thanks!)
• Adds more of a mental burden to client authors learning about earthstar (boo!)
• And confuses potential pub operators (argh!)

That's not what I want to contribute to this project.

I started building earthstar-graphql from the perspective of a client author who would use it to build complex apps very quickly using a rich existing ecosystem of tools.

But after having built some things that talk to each other, I now see that a huge part of earthstar-graphql's value is in being able to query and mutate earthstar peers using a common query language over any transport: HTTP, websockets, function calls, messages delivered by pigeon.

This is why I think it could be a good solution for RPC.

Because earthstar-graphql is supposed to be able completely power an earthstar client, this means you can do a lot remotely: getting authors, workspaces, documents, triggering syncing, setting data to workspaces, etc.

And this made adding syncing to earthstar-graphql very easy: you query another earthstar-graphql instance's documents the same way you would locally, and then ingest the resulting JSON into a IStorage.

It also means there are possibilities for things like proxy stores, which @cinnamon-bun saw the potential for pretty much immediately, and made this diagram

---

What I'm proposing is adopting GraphQL as the way to communicate with pubs. Most importantly, I think we can do this in a way where no-one needs to know anything about GraphQL.

The core earthstar package already does this with REST endpoints, where the implementation of syncing is abstracted behind syncLocalAndHttp. Users don't need to know what the routes and methods are for fetching and mutating resources, they just call a function and earthstar does it for them.

The same thing could be done with GraphQL, where users simply give an IStorage and pub URL, and the sync happens without anyone needing to know how, just like syncGraphql does.

It wouldn't add any dependencies. All we need to do is send a query string over HTTP.

What we get from it is being able to easily add new capabilities to syncing (and I think Earthstar's syncing is going to be powerful!) without having to maintain our own parallel vocabulary of REST endpoints and all the possible params that can be given to them, because we'll have a powerful way of doing that at our disposal already.

There are other benefits, e.g. because GraphQL can support streaming updates, we can do things like continuous real-time syncing.

This is asking a lot, and I can completely understand if you'd say no to it. Earthstar wisely wants to use boring tech, and REST is more boring that GraphQL. But I feel like GraphQL's no spring turkey either. It's been five years since being released publicly and has a lot of documentation, tooling, and adopters.

People wanting to write their own earthstar libraries from scratch would need to learn a little about GraphQL, but the same is true of learning all the REST endpoints. But they could also copy and paste the GraphQL queries used by earthstar in their own projects.

Otherwise, I think we'd be best off having one kind of pub, and one kind of syncing — and I think it'd be best for Earthstar if I removed the notion of doing these over GraphQL.