Can I do symmetric peer-to-peer?

[a-type]

I'm curious about using this tool to achieve a symmetric sort of peer-to-peer pattern, where both peers are 'equals' and can invoke RPCs on one another at will.

The design of the library seems to be more of a server-client architecture, at least in terminology and basic usage.

Theoretically, I could have each peer host a server instance and also connect a separate instance as a client to its peers, achieving bidirectional server-client relationships on all edges. That seems like a good bet, but does this also open up two channels of communication for every peer relationship? How does that scale?

Is this a good way to approach this problem, or would you recommend attempting to continue in the server-client model but rely on messages and replies to achieve some sort of symmetric communication? The blocker there appears to be that only the client peer can initiate an RPC and the server cannot invoke RPCs on clients, is that accurate?

Or is this library simply not oriented toward that use case and would be a bad fit?

[chr15m]

server cannot invoke RPCs on clients, is that accurate

No, that's not accurate. You can accomplish this by simply passing the other party's pubkey into the rpc/send call at both ends:

b.send(otherpubkey, message); # send a simple JSON message
b.rpc(otherpubkey, rpcfunction, args, callback); # initiate an RPC call on peer

At the time when I wrote Bugout I got super excited about the idea of client-server architecture over WebRTC but looking back on it what people probably want more than that is a good robust p2p implementation with all clients on equal footing. simple p2p networking for the web browser.

Maybe I should change the README to highlight that use case more clearly, and have more examples like that. What do you think?

[a-type]

It might be helpful to have some more concrete use cases, but issues can also serve to document, which is part of the reason I asked!

Thanks for the tip about specifying an address directly - I saw that in the source but didn't realize the implications. That makes sense. I noticed all current peers are stored in connections so I'm starting to piece things together.

I appreciate the response. It seems like I could pretty easily build an abstraction around this to create a more p2p oriented usage pattern, so I think I'm good to go.

[a-type]

Actually, coming back to this today to build a more p2p-focused prototype, I'm still a bit perplexed by the client-server constructs in the usage and how they could be adapted to a more symmetric pattern as-is.

For example, suppose I ultimately want to end up with a mesh of peers who are all connected and invoking RPCs or passing messages to one another at will. In order to establish relationships between peers, I want to do a simple connect:

1. Peer A wants to connect to Peer B. B sends her address to A in some out-of-band channel, like via SMS.
2. Peer A now wants to add that address to his peers. This is the part I'm not sure how to do. According to the docs and my best reading of the code, the right way to do this is to instantiate a new Bugout(addressOfB). But A already has his own instance of Bugout({ seed: aSeed }). Suppose we do it anyway...
3. Peer A now has 2 Bugout instances: his own, and one which connects as a client to B. But B only has 1 instance, her own. They can now communicate, but this disparity seems pretty arbitrary and makes it complicated to manage on A's end.

I think what I would need in this case is an alternative usage pattern exposed by the library for doing a truly peer-to-peer model - mainly, the ability to arbitrarily add any public address to the peers of my pre-existing Bugout instance. Suppose 2 above becomes: "A calls bugout.addPeer(addressOfB) to add B to his peers." That method does any initial handshaking and plumbing to connect to B just as if B had established a client-server relationship with A herself. Now our two peer clients have a symmetrical configuration.

I might write a bit of a wrapper to do this manually for now and see if it works, as I'm getting a bit of a handle on Bugout's source. Looks like I could manually register a value in peers and then send a ping to the peer to announce myself, maybe?

Could be I'm just forcing a square peg into a round hole here. You're the one defining the use cases for this library, so it's up to you if you think this is a good pattern to support here. Otherwise I may end up using your work as inspiration for something home-grown, which is totally fine.

[draeder]

I was playing with returning a list of connected peers yesterday and used let peers = b.peers() which returned a JSON object with peer addresses along with their ek,pk key pairs -- that may be of some use. A better way though is to use b.on("seen", function(address){ //your code to handle peer addresses }). Either way, I'm interested in any approach to establish symmetery also. One of the challenges I've contemplated is how to retain a remote peer address after they've disconnected and reconnected in order to persist their "identity" for purposes of saving a list of "followers". An instance's peer address changes every time it reconnects. However, the remote peer seed address can be stored and reused, which is how the server is reachable. Perhaps the remote peer seed address could be used? I'm not sure.. It's reachable in the local browser instance through let seed = b.seed()

With that said, the approach I've been playing with has been to use bugout to create a "popup" peer network. The use case would be to pass a URL with the server address or peer swarm ID as a search parameter. Let's say the protesters in Hong Kong want to communicate. Someone could print out stickers with a QR code tied to that popup network. People scan the sticker with their phones and are all connected to the same server address or peer swarm.

[a-type]

@draeder FWIW, I believe peers should have stable addresses if you save and restore their seed yourself (for instance, using localStorage). That's discussed in the README. I think that would give you persistent relationships if you store the addresses of the peers you connect to. The problem is, how do you reconnect to those peers (if they're available) when you restart the app later? That's my issue; it seems like you'd have to manually construct a new Bugout instance for each stored friend address.

I'm still reading through the code to determine what changes could be made to enable more seamless peering via a single instance.

[draeder]

@a-type Apart from the issue described in #29, I have not had any problems re-establishing peer connections regardless of if the 'server' is disconnected and later connected, or the client. As for stable addresses, I have been experimenting with console.log(b.address()) (which is the local peer address that is passed to the server) and it does indeed change each time the connection is re-established. I could easily be missing something here though as I'm just a javascript tinkerer and have much to learn.

Edit: from some of the under the hood looking that I've done, the seed is what is passed to WebTorrent as the unique identifier peers use for signaling to establish the WebRTC connection. From there every peer has a unique address for the purposes of communicating over WebRTC. Maybe I'm confusing seed and address. I'll have to try sending messages using a peer's seed address from b.seed and see if that works. If so, that might be the answer.

Update 1: Using the seed of the remote peer results in a failure, and I believe that is because only the server's seed is used as the channel identifier in WebRTC:

Uncaught BohM8UybiCmd8eA6NWTY6v9KmcTjdgZ26wKeFjV3ACMb1raCLcwg not seen - no public key.
n.send @ bugout.min.js:1
(anonymous) @ spsn-api.js:54
l @ bugout.min.js:6
r.emit @ bugout.min.js:6
i @ bugout.min.js:1
(anonymous) @ bugout.min.js:1
_onExtended @ bugout.min.js:1
_onMessage @ bugout.min.js:1
_write @ bugout.min.js:1
h @ bugout.min.js:1
f @ bugout.min.js:1
l.write @ bugout.min.js:1
i @ bugout.min.js:1
s @ bugout.min.js:6
r.emit @ bugout.min.js:6
c @ bugout.min.js:1
l @ bugout.min.js:1
d.push @ bugout.min.js:1
_onChannelMessage @ bugout.min.js:11
_channel.onmessage @ bugout.min.js:11

Here is the relevant code if you're interested:

   let b = new Bugout(identifier) // getting identifier from localStorage or address bar in code not shown here

    // Detect connected peers
    b.on("seen", function(address){
        b.send({seed: b.seed,address: b.address()})
    })
    b.on("message", function(address, msg){
        console.log("Remote seed: " + msg.seed)
        console.log("Remote address: " + msg.address)
        b.send(msg.seed, "Got your message using your seed address to reply")
        //b.send(msg.address, "Got your message using your local address to reply")
    })

Notably, using commented out line to use the peer address also fails, even if I use address instead of msg.address. So now I'm really confused 😕.

Update 2:

Something wasn't quite right with my code for using address. This code works. Note that it will create a loop.

    let b = new Bugout(identifier)

    // Detect connected peers
    b.on("seen", function(address){
        b.send(address, {seed: b.seed,address: b.address()})
    })
    b.on("message", function(address, msg){
        console.log("got message from: " + address)
        console.log("Message: " + msg)
        b.send(address, "Got your message using your local address to reply")
    })

Anyhow, it brings us back to square 1, though: peer addresses are not static between reloads. Fundamentally I think this is going down the path of identity linking, moreso than p2p symmetry. @chr15m wrote a pretty succinct article about identity linking here. Hopefully, my comments are helpful even if I may be misunderstanding things..

[a-type]

Oh, I see. Yeah, seed just appears to be an arbitrary piece of data from which the Bugout client's identity key pair is generated. Then the pk value is generated from that pair's public key, and then the .address() is encoded from that value.

In order to keep an instance's .address() stable, you have to construct the 'server' with the seed - const b = new Bugout({ seed }). Note that's not new Bugout(seed) - that's for a client connecting to a specific server. To start a server with a specific and consistent seed (and therefore public key, and therefore address), you pass in an options object instead with a seed key.

I haven't tried it personally so maybe you have and already ruled it out, but that's how it looks like it should work based on the source.

[draeder]

I think this question is similar to what you're asking for, too: #15. Although I've since abandoned doing things this way since the power here is a WebRTC channel that can be used by anyone with the passed in identifier (server address for client-server, or any string for a peer swarm). My focus has been more on how to maintain a persistent identity of connected peers, which is an entirely different challenge I suppose.

[a-type]

With a bit more digging I think I have the trail to my answer.

What I was trying to figure out is how a client connects to a server. If I can determine how that connection is initiated, I can figure out how to cause an existing client to connect to some peer(s) arbitrarily without having to create new instances.

Well, that doesn't seem to work with the current architecture, but at least understanding it will help me figure out if I can repurpose some of the ideas here.

What I've figured out:

1. The "Server" instance starts seeding a torrent which takes its name from the server's address.
2. "Client" instances do not seed their own torrents - they seed the server's torrent. I.e. they seed a torrent with the same name and the same data.
3. WebTorrent appears to do the rest - it connects the seeders together since they are referencing the same torrent with the same content.
4. When they connect, Bugout utilizes an extension on the BitTorrent protocol to begin passing arbitrary messages between peers seeding the same torrent.
5. To leave a peer relationship, you just stop seeding that torrent.
6. To drop all connections as a server, likewise, stop seeding the torrent.

I think it's clicking. And this is definitely an inherently asymmetrical connection pattern.

I have to think more about what I really want though. I'm playing around with the concept of a decentralized social network. What I probably want, in that context, is actually not truly symmetrical peers - but for each user to be their own 'server' and also connect to all their friends.

Seeding a bunch of torrents at once obviously isn't crazy...

I might just need to write my own client, either wrapping multiple Bugouts, or just Bugout inspired, which manages all this multichannel stuff internally.

[draeder]

@a-type That sounds about right and is along the lines of my initial thinking about how to use Bugout. I've been pondering the notion of a decentralized social network protocol for many years, which is what social networking should have been from the beginning (like email, in a way). If you do build your own Bugout inspired API or client wrapper, please come back and comment with a link to the repot. I'd love to see it in action.

[a-type]

@draeder I took some time today to transcribe Bugout into TypeScript in my exploratory project and make modifications so that it creates mutual relationships with peers by "following back" (i.e. seeding a torrent with the peer's address) when they connect. By doing so I was able to make a simple peer-to-peer chat demo pretty quickly! It's not hosted though, just code for now.

https://github.com/a-type/cnxn/blob/bugout/src/p2p/P2PClient.ts

Most of the code there is more or less 1:1 with Bugout, although I made a lot of changes for clarity w.r.t variable names and of course adding full typing. I also dropped RPC support since it's more or less a layer over generic messages and I'm planning to do that in a higher abstraction.

The biggest new change is the followTorrents class property which stores the torrents created for each peer the client is following. During the onSeenPeer part of the flow, the client will follow back peers it hasn't seen in a while.

There's a lot still TODO, including cleaning up those follow torrents when the peers disconnect. But it's something.

[draeder]

@a-type any chance you could browserify/minify this so I can play with it via a script tag directly in the browser? I wonder if @chr15m would consider some of your modifications for an official commit...

[draeder]

@a-type Do you really want to follow back every peer who was "seen" at some point? Maybe I'm misunderstanding how that works.

[a-type]

@draeder I'm still figuring that last bit out. Right now I'm presuming that every peer that begins seeding "your" torrent (the 'server' or 'identity' torrent) is trying to connect with you as a friend, so it makes sense to instant follow-back. On the flip side, no 'random' folks should start seeding your identity torrent since they wouldn't know about it.

That's a pretty big assumption and also tied heavily into my completely symmetric use case. This is not so much a Twitter idea (following, 1-way) but more of a Facebook (where all relationships are 2-way).

I haven't used Browserify, but I can look into it maybe?

[draeder]

@a-type So as Bugout works now, peers can be connected indirectly through other peers. That's something to be aware of.

As far as browserify, I could probably set it up myself and make it work, but I'm a humble javascript hobbiest :) ... One thing that attracted me to Bugout is that I can use simple html, css, and javascript to do everything with a simple script tag in my HTML doc.... I have avoided react, angular and vue like the plague because they all require servers. I want true p2p. Bugout gives us that...

[chr15m]

Hey, just wanted to clarify a couple of things.

The .seed property should be kept private to the client who is using it. As you figured out, it's used to generate the public/private key pairs which are used for encryption and authentication. Don't share the seed or nodes will be able to hijack each other's communications.

The basic connection model for Bugout is channel/room based. To get multiple nodes talking together you connect them to the same room. When you connect to a single Bugout node using its address, what you are actually doing is connecting to a room where the room name is just the pubkey of the other node. Bugout privileges this kind of connection slightly by making that address the default when doing RPC, but other than that it's much like connecting to any other room. So it's all about rooms using a common connection identifier (first argument to Bugout).

Under the hood what is actually happening when you connect to a room with multiple people in it is each peer in the room is making a connection to each other peer in the same room. Same as when you share a torrent. There is a limit on this and so every node also forwards all messages it receives to all the nodes it knows about (gossip protocol). That redundancy means that there can be gaps in the connectivity graph of peers in a room (e.g. if two peers can't see each other because of firewall or whatever) and messages still spread to everybody in the same room.

Hope that clarifies things a bit. @a-type I think I need to understand your specific use-case a bit better in order to comment on what you're trying to achieve.

I've thought about the "social media" use case quite a bit too and I always imagined it like this:

• Your "identity" would be a pubkey.
• When your device comes online it creates a Bugout instance with your pubkey as the room name (the default. e.g. you become "server").
• You also serve a webtorrent containing all of your posts.
• Your device connects to the pubkey-rooms of the people you follow.
• When people connect to your pubkey-room you send them a message containing the info-torrent hash of all of your posts.
• When you post, you send a message to everyone in your pubkey-room a message containing the new info-torrent hash of all your posts.
• When you connect to somebody else and they send you their info-torrent hash of their posts, you grab the torrent, seed it, and start downloading and displaying their posts.

Replies and likes etc. are a bit more complex but also doable.

The nice thing about this architecture is friends can share your info-torrent and hang out in your pubkey-room even when you are offline. So if you're offline but somebody connects to your pubkey-room a friend can send a message with a copy of your last cryptographically signed update message with the info-hash of your posts, and the new person can download your posts from your friends who are also seeding them.

I will try to write this up properly at some point. Seems like "decentralized social network" is something a lot of people find valuable.

[a-type]

@chr15m That sounds exactly like what I'm building. I have essentially forked the Bugout code to orient it more toward managing multiple "room" torrents (what I've called "identity" torrents) for yourself and your friends. I'm currently just trying to think through the implications of how to ensure data security while also maximizing use of the redundancy properties you mentioned.

I have a specific set of design constraints which I came up with even before I found this project or considered torrents which line up really well with these implementation details. I'll probably write up my design a bit more on my repo (https://github.com/a-type/cnxn) when I get the chance. I'm working on a bittorrent branch at the moment.

[draeder]

@chr15m Thank you for the clarification (this has helped me see some things I'm not doing right), and @a-type thank you for bearing with me (thank you both for that, actually). Javascript is a hobby for me and not quite my day job, but I'm learning more all the time thanks to the ability to have these types of conversations. I was able to quickly solve a javascript problem at work the other day thanks to conversations like this.

[a-type]

@draeder happy to be of help. JS isn't everyone's passion, which is fine... it requires a lot of headspace to stay up to date on that ecosystem. For what it's worth (and off topic), React and other frameworks don't require a server either - they compile down to plain old JS files you can put in Github Pages or S3 or wherever. Just in case you ever want to dive in a little deeper, you don't have to compromise on being truly p2p.

On topic, I've actually shifted my focus back to OrbitDB / IPFS for the time being. I didn't understand the patterns when I first looked at OrbitDB and I got intimidated. Spending some time with Bugout (and rewriting it from scratch) really helped me grasp how p2p works, and for that I'm very grateful! OrbitDB seems a bit more robustly oriented toward true peer relationships, while keeping a lot of the same concepts - for example, Magnet URI = Multihash, Bugout 'rooms' = PubSub channels, and of course Torrents are replaced with IPFS hosting.

[draeder]

@a-type Check out the relatively new AvionDB. It's an abstraction built on top of OrbitDB that brings a syntax similar to MongoDB, which makes things much simpler.

[draeder]

Closing, as this issue appears to be resolved and clarified.