Allow wildcards in allow list

[basham]

With the old Earthstar Pub, any workspace could be synced with it. With Replica Server, it takes an understandably more restrictive approach, via allow_list.json. However, it would be helpful to have some solution to allow additional share addresses to be included, without having to manually update this server configuration.

In my case, I'm building an app in which any number of shares (workspaces) could be created on the client. The addresses are dynamically generated on the client, so there isn't a mechanism to know what they are up front.

Perhaps the allow list could permit wildcards (*), so that the server admin may opt in to a dynamic list.

For example, you could allow any addresses that are prefixed with "+namespace.":

[
  "+namespace.*"
]

Or even just allow any addresses to sync with it, like the Earthstar Pub.

[
  "*"
]

[sgwilym]

I'm afraid this will not be possible as you've requested it: when Earthstar peers sync with one another, they initiate with a 'salted handshake'. They both share a salt to hash with, and exchange hashed versions of the addresses they hold. If anything matches, they know they can sync that share. What they can't do is say "I'll take anything beginning with 'x', or anything at all".

This has quite a few benefits: peers can't accidentally reveal shares to each other, and you can keep a single list of replica servers for all your shares.

But I do think we need to accommodate situations where share addresses aren't known up front.

Could you tell me a bit more about your application? How were the dynamically created addresses being shared with other peers in the old setup?

[basham]

Here's the workflow as I have been using it with Earthstar v6 in a web app:

1. The user creates a workspace (share). They provide a name for it (but this value is saved in a document within the replica). They provide a pub link (replica server URL).
2. The app generates the share address, in the format of +namespace.id. "Namespace" is the app's name. "Id" is a generated id from cuid. This address is mostly obscured away from the user as an implementation detail. (For example, I may not care about the particular ids that are associated with a YouTube video or a Facebook event, as long as its unique and it works.)
3. If the user wants to share this workspace with someone else, the app provides a join link.
4. This join link points to the app, with a join code as a parameter.
5. This code is a base 64 string containing the address ("id"), name of the workspace at the time that the code was generated, and the server URL (btoa(JSON.stringify({ id, name, server }))). It is meant to be just obscure enough to discourage users from manually tweaking it, but it's not a big deal if a savvy user would figure it out.
6. This same join link is the result value of a QR code on the same page.
7. The user can either copy the join link and pass it along to the other person. Or, the user can have the other person scan the QR code.
8. The second person decodes the join link. The name is used just as a user-friendly confirmation that they are looking at what they expect. The user confirms they want to join. Then the address ("id") and server URL are used to initiate the workspace and start syncing, pulling down the latest data.
9. Once the data is pulled, it sets this new workspace as the current workspace (because a user can switch among any number of them on that device) and redirects to the landing page of that workspace.

Realistically, there won't be many of these workspaces created. So, even if I have to manually append to the allow list in the meantime, that's acceptable.

However, it would be nice if there was some invisible mechanism for the replica server to add to its allow list. Maybe that would involve some additional handshake. Maybe the replica server could provide some keypair or whatever that admins can distribute to those they trust and who wants a space to store their shares. Maybe this code has some sort of one-time use or a limited number of uses or a time out. Perhaps that code could be like the join code I just explained. It could have in itself both the server URL and some public authentication. On the user side, they just see it as some sort of special code. But it provides all that's needed for it to point to the right replica server and have it trust the source enough to append to its allow list.

Another idea is to have the replica server generate its own share addresses upfront. Then admins can distribute them to who they wish. But if a share is already created on the client, how do you transform that share address into another, so that the client and server has the same address? Or, is there some means on the client side to map their local addresses into a remote address, or have the server map client addresses into addresses on its side?

I'm just guessing here, though. There are a lot of security concerns that I have little expertise in. And I'm just brainstorming. I'm open to whatever.

[sgwilym]

Now that replica servers are extensible, it's possible to create an extension which allows you to add new shares to a replica server via a network request. I'm not sure whether I'd make this a core extension, but it's all in userland so you don't need me anyway!

@basham What do you think?

[basham]

This extensions idea is really neat! Thank you for going above and beyond with this feature.

I'm studying the source code and planning. The ExtensionKnownShares looks to be a good template for me to follow. I don't think it'll work for my particular use case, as-is (which is of course okay). It instantiates replicas when the extension registers, and there's no mechanism for doing it during runtime, via the hander. So, for me, I can create a custom extension, have some way for a client to request a new share address via the handler, and when that happens, initiate a new replica and somehow persist the address to disk.

I agree, I don't think this is appropriate as a core extension. The mechanism for requesting (and authenticating) a new share is very much user-land territory. I could see this custom extension being included as part of a new example in the examples folder. Once I get around to making this work on my end, I'll share it with you (maybe a gist?), and you can choose if you want to do anything more with it.

I like it. I think it'll work. Feel free to close this issue. If I run into trouble with this direction, I'll reach out or open new issues as needed.

[sgwilym]

@basham Rad. Thank you for exploring this, I think replica servers will need some form of supporting unknown shares and it'll be great to find how these patterns play out.