Discovery swarm API

[RangerMauve]

A lot of P2P use cases require some sort of DHT or discovery process.

From looking at the the community, the DatArchive API is good to solve basic data access and saving, but peer discovery is still an important part of applications where you want people to communicate with each other without knowing about each other beforehand or having centralized services track peers.

• https://github.com/cabal-club/cabal
• (Try it out with npx cabal --key dat://21b2b9ff201b01e6081709d82e6b81a5cf3a68d2cd5f092d0ffec58772642892)
• https://github.com/substack/chatmesh
• The various things Jim Pick has been doing with Tiddlywiki and the shopping list demo

All of these are essentially replication logic over top of peer discovery through discovery-swarm.

I think @pfrazee Mentioned some sort of PeerSocket API idea, and after playing around with Cabal and thinking how it'd be impossible to get something so simple (relatively speaking) to work in Beaker, I think it's really needed to make the P2P web more flexible and more powerful.

I don't think that WebRTC data channels are going to be good enough for p2p since they still need centralization to work and consume a lot of resources per connection.

I doubt browsers are ever going to be too keen on having web pages set up TCP or UDP servers, either, since it's very low level and will be hard to secure.

I think an abstract DiscoverySwarm API that lets you find peers for a given key, and being able to connect to them and get back something that behaves kinda like a Websocket or RTCDataChannel will be high level enough that it'll be useful for developers while still allowing browsers to constrain the page. Namely it'd need onopen, onmessage, onerror, and onclose, as well as send() and close().

Although this would obviously be super useful for the Dat ecosystem, I'm sure it would get rid of a bunch of complexity for other p2p protocols and help grow a healthy ecosystem by getting rid of the hardest part of p2p.

Though using the current discovery-swarm features would be useful, I think that in the future it would be useful for browsers to integrate with stuff APIs like Wifi Direct to discover peers in physical proximity. Thus applications can code against an abstract "Discovery" protocol will have a simple high level API while supporting every type of p2p application from a totally offline local mesh network, to discovering peers over the bittorrent DHT.

[pfrazee]

I agree. This is what I was referring to when I said we're planning to add either PeerSocket or some kind of WebRTC signaling solution. I'm leaning toward the latter in the short term (some kind of discovery & signaling API combined with WebRTC) but we'll see what happens.

[RangerMauve]

I would really advise against using signaling since it won't let us build fully decentralized or offline-first apps.

Would a PR be welcome with a discovery API? Or even a mock-up of what it could look like to prove its usefulness?

[pfrazee]

Signaling doesn’t imply a server. We can signal with a dht

On May 25, 2018, at 9:07 AM, RangerMauve notifications@github.com wrote:

I would really advise against using signaling since it won't let us build fully decentralized or offline-first apps.

Would a PR be welcome with a discovery API? Or even a mock-up of what it could look like to prove its usefulness?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[RangerMauve]

Sorry, I assumed you meant the usual signaling mechanisms being used for WebRTC, my bad.

Will the DHT be available as a JS API in the browser, or are you planning on a more high level API that does the signaling behind the scenes?

Why not make the DHT available as is and let users do signaling for WebRTC themselves?

Even if the signaling is done through a DHT, this won't be compatible with existing ecosystems that make use of TCP for connections. If the browser abstracts away setting up the server for incoming connections and for connecting to peers, then it can support existing applications while still hiding information from the client-side.

[RangerMauve]

Sorry, I think I'm rushing too much. 😛

What's the goal with your WebRTC / DHT signaling project? Is it going to be integrated with Beaker? Is there something I can read to learn more aobut it?

No rush in responding by the way. I'll see if I can stalk around the various forums for more info.

[RangerMauve]

To clarify my reasoning:

WebRTC provides really nice defaults for P2P since solves a lot of issues in p2p networking. It uses DTLS and is thus end-to-end encrypted, automates discovering local addresses via ICE, and supports standards for traversing really difficult networks through TURN relays.

However, this also means it's harder to get WebRTC adpoted. It provides a lot more overhead for implementations and there are no environments support for it out of the box other than browsers.

My idea with bringing discovery swarm to the web is to have the simplest possible API to integrate with existing ecosystems and to make it easy to develop new ones. Searching for peers over the Bittorrent DHT and MDNS is based on standards that are likely to be available in most environments.

I believe that although WebRTC will have a better payoff in the long run, once implementations are more optimized and widespread and there's more widespread use of it, giving access to existing ecosystems will be better for adoption in the short term and will help the technology (and overall p2p mindset) spread to people much faster.

Having the ecosystem switch to this new WebRTC-based transport will have a lot of fragmentation and will make it harder to reuse existing networks and existing dat archives (or other hyper-* data structures).

A standard for signlaning for WebRTC over a DHT will be great for WebRTC-specific applications, but I think it will take a lot longer before we see the payoff, and it won't help spread applications being built in the dat ecosystem today.

[pfrazee]

EDITED for clarity

@RangerMauve I agree with a lot of your points. It's pretty weak that WebRTC can't be used outside the browser, and it's also not super performant in browsers.

I'm probably not being clear enough, because I agree with everything you're saying. I just want to add a way for users to establish data channels with each other, completely separate from Dat's mechanics. That involves two pieces: 1) a discovery system and 2) a connection system. I agree that we should expose Dat's discovery system in some fashion to solve that first piece. For the second piece, I'm leaning toward using WebRTC for now because it's the least opinionated choice.

So what I'm actually suggesting is the most minimal set of changes possible: adding a discovery API that can hook up with existing WebRTC APIs, and making no change to Dat.

[RangerMauve]

So if I understand correctly, the goal with this WebRTC enablement isn't to interoperate with the existing Dat network, right?

Do you have any details on what the discovery system / siganling would look like? Or is that just in your head for now until there's time to actually work on it?

Would you if I participated in the brainstorming for how this will work?

[pfrazee]

So if I understand correctly, the goal with this WebRTC enablement isn't to interoperate with the existing Dat network, right?

Correct, it's to establish general-purpose data channels with other users

Do you have any details on what the discovery system / siganling would look like? Or is that just in your head for now until there's time to actually work on it?

Not yet

Would you if I participated in the brainstorming for how this will work?

I'll keep everyone updated but it's going to depend on too many internal variables for us to externalize yet

[fsteff]

Is there a chance we get access to discovery-swarm in the near future?

This would allow apps to use plain hypercore/hyperdb. Would be cool to have something like discovery-swarm-stream included in beaker too.

[RangerMauve]

The proposed "ephemeral channels" thing got shot down at the WG meeting due to privacy concerns.

Access to discovery-swarm might raise the same privacy concerns.