True P2P

[worldpeaceenginelabs]

Hi Draeder,

i asked "I am working on a True P2P project (without fixed signaling servers) Does anyone have experience with a combination from GunDB, webtorrent trackers and dyndns, maybe???" in the Gitter amark/gun chat.

Rococtz answered, that you have been trying something like that for a while now.

My definition of True P2P = "A swarm of P2P clients, independent of any dedicated servers, not even dedicated signaling servers"

If i did not super misunderstood this repo https://github.com/subins2000/p2pt, then it will make available the webrtc datachannel for your app, which are already established by webtorrent for file transfer purposes (i guess this is how they do it).

The webtorrents, websockets, bittorrents are free to use, so we are not talking about me, building a relay network, but using bittorrents relay network (75.000) for my p2p swarm, in the same time improving the bittorrent network (more nodes, better scale, better performance/reliability)

The further improvment is to have multiple such entrypoints (free signaling servers) like DHTs of free stun and turn servers, multiple dyndns trackers as fallback option and so on...

Anything wrong in my thinking? Maybe I cant use the trackers for free? I am really not sure, even though reading multiple blogs. But if the trackers are free, it works out, right?

Is there any of your repos already doing this or in development to do so, coincidentally???

This is the repo i am working on: https://github.com/worldpeaceenginelabs/CLOUD-ATLAS-TRUE-P2P

[draeder]

Unfortunately, no P2P network can escape the need for bootstrap servers/nodes of some kind, and with browsers, signaling servers are a must because browsers can't listen for incoming traffic. The problem with WebTorrent is the scarcity of tracker servers, which are the signaling servers for WebRTC. They are commonly overloaded. Gun is working on solving this for the Gun network with its AXE protocol. I can see AXE as a way to eventually decentralize WebTorrent tracker servers.

Easypeers works pretty much the same way as P2PT. It creates a torrent that peers swarm around and connect to each other with. But as with P2PT, Easypeers also requires a signaling server for WebRTC connections. And, of course, it isn't finished yet.

You might want to look into simple-peer, which is absent any signaling server, so requires some way to exchange WebRTC SDP. It is what WebTorrent uses under the hood, and it makes WebRTC pretty simple once you understand how it works.

The bittorrent-tracker library is free. You can run your own tracker server with an easy 1-click deploy to Heroku with p2p-tracker.

I have a domain name I would be willing to contribute to anything you come up with from a DNS perspective. Just ping me on twitter and we can work it out.

Hope this info helps

[draeder]

Speaking of 'Atlas' that makes me thing of MongoDB Atlas. I think I might whip something together with change streams for WebRTC signaling using that.

[worldpeaceenginelabs]

Thanks for the answer. It helps a lot understanding the basics.

Axe is definitly interesting, but for me the concept is already "too capitalistic" and too monopoly. I dont want to have to pay to go online, i just want to go online :D Except i am not getting the concept and there is no cost or wallet connect or similar for the app developer or their users involved. If usable for free, you could put AXE on top of the stable of strategies to reconnect.

I know i am counting peas, but my goal is independent or minimum quasi independant P2P.

Which means, not dependant on one single network infrastructure, not my own, not a foreign one, but allowed to use every chance to connect as a fallback to reconnect. (in turn using different, already existing network infrastructures as needed. at the end of the day, we want the connection to the swarm, to update our DHT trackerlist/peertrackerlist/peerlist, to STAY connected)

What about this MSC MEP concept then? (Multiple Strategy Concept, Multiple Entry Points)

Having multiple strategies -jamstack decentralizes the centralized "very first entry" domain -pwa gives it offline capabilities out of the box -dht lists saved on free github page, blogs, forums (updated in an interval, automated via script on clients, sharing the updated dht to your page, a free blog for this purpose only, in a forum, etc., everything thats free and allowed, ergo not stresses out the free tier)

PLUS

using multiple entry points -tracking/signaling (using multiple protocols for entry points) in a priority list with timeout limits (p2p first, then fallback options, worst the "very first entry" domain)

I imagine it happening like this

This is the users client on the "very first entry" domain, which is live for demo: https://cloud-atlas.app/ (only the Google Login is active) The client is a JAMStack website (static html file, which is hydrated via API, stored on GitHub, build and served from Cloudflare Pages(200 CDN server), at zero cost) Its a PWA, so it can be installed like an app, running your website/app itself even with https://cloud-atlas.app/ offline. Overall, it behaves like an app, which is accessable through a website.

Every client is also a tracker itself, so its able to fetch connection adresses (DHT)

1. direct (webrtc) and indirect (torrents) (webrtc prioritized, Peerservers but with webtorrent AND bittorrent)
2. direct tracker-swarm
3. Easypeers or P2PT to reconnect via webtorrent as signaling server, than back to 1 & 2

Summary

So this is using two different ways taking advantage of the webtorrent/bittorrent network first, (Peerservers and Easypeers/ P2PT) but as soon as you have enough peers continously, these already online peers are becoming the 99% of the time used p2p trackers (some time after their launch, for instance Facebook, had minimum a few users online every second of a 24h day, which is the equivalent to 99,9% server uptime in a p2p network)

Fallback 1

Besides exchanging their DHT steady, they also are able to get their dht via a classic torrent, saved on some torrent websites, or like a post somewhere with a magnet link you save every few hours automated on a/or multiple socialmedia websites, and god knows where else...

???Fallback 2 and possible "mainnet"??? (here mainnet is the most reliable fallback option which would equal a centralized server) The swarm always found via dyndns???? Maybe Bugoff as a decentralized P2P API server(appdata & dht), always found via dyndns???????????

Fallback 3

Your "very first entry" domain/site gets a 60sec updated dht list. But only used as last last last fallback, after everything else what i just wrote failed.

You get the upside down concept, which causes pragmatically what we could call True P2P territory, right? ;)

These beasts (clients) pretty much crawling their options for connections, the opposite way a centralized entity would do. 🥳

Would that work? Are my peers steady online now, independant from fixed infrastructure, which becomes just the fallback in case, or did i miss something out?

You know whats cool? We are just about to figuring out how True P2P could work out. Which has the power to transform the internet from ground up. (so lets keep it open-source) Thats minimum so cool as figuring out True AI (like from the movies) ;D

[draeder]

Just a quick note: AXE is part of Gun. All Gun modules/protocols are free & open source.

@amark's ideas with AXE are very similar to some of those you mentioned about storing relay addresses on Github, elsewhere, etc.

I do like the idea of one p2p technology being used to make another one more robust. Tristero does this to some degree with IPFS, WebTorrent and Firebase.

[worldpeaceenginelabs]

Ok, then AXE definitly gets interesting as soon as its available. And Wow. This Tristero repo reads awesome! A nice extension to the above described concept.

Well, do you see any struggles, shortages, narrow pipes in the concept? Are there any obvious development struggles involved?

[draeder]

Are there any obvious development struggles involved?

Considering I have yet to be successful with my 5-6 year long quest to decentralize tracker servers, yes there are many development struggles! :)

Curious to see what you come up with

[worldpeaceenginelabs]

Well, i have all your repos available plus the ones you suggested, plus my own research.

I am pretty new in coding, so i cant really estimate, if having all these libraries available is a serious advantage? Are you?

For me, it looks like puzzling it together with these libraries and than finetuning, finetuning, finetuning... or not?

[worldpeaceenginelabs]

We switched the repo and i found a collaborator already.

You can follow us on https://github.com/worldpeaceenginelabs/True-P2P, which is based on Trystero, and now we are adding modules (first is webtorrent)

Thanks for your suggestions. It pushed the project forward a lot, making it much easier to realize (and finally find a starting point)

[draeder]

Nice! Great to hear! I'll certainly be following along

[draeder]

Oh man, I forgot about some of those old repos. They mostly represent failed attempts to get things working. I think tracker-swarm is probably the most useable of the bunch; but I don't remember where I left off with it. I believe it works as described in the readme.

[worldpeaceenginelabs]

Hi Draeder. Could you help me out maybe? It turned out, that Trystero is using webtorrent already (i was confused by him using the bittorrent term)

I like to add two of your repos as modules to my repo. tracker-swarm, and webtorrent-beacon like you used it on your peerservers repo.

If i understand webtorrent beacon right, you could potentially do the following with it:

My idea is to use a torrent as a beacon. So your app only has the name /signature of a torrent saved, which nobody knows about, except your app (like a secret). Everybody who is interested in this torrent (only people with your app, nobody else knows), is potentially one of your users, so you connect to them.

For security, you could exchange this torrent from time to time, which is like changing the secret address or password from time to time.

[draeder]

Take a look at the Easypeers code. It uses WebTorrent extensions, which will probably work better for you for that purpose. It's basically what P2PT and Bugout does as well. It creates a torrent based on a 'topic' and peers interested in that topic swarm around that torrent and connect to each other.

[worldpeaceenginelabs]

so easypeers replaces tracker-swarm and webtorrent-beacon at once?

did you use these two repos/concepts to realize easypeers?

Sounds like it does exactly what i just wrote? "My idea is to use a torrent as a beacon."

[worldpeaceenginelabs]

Which one is more developed? (for production)

Easypeers or P2PT?

[draeder]

P2PT. Easypeers is a work in progress. I have to finish the gossip protocol.

[worldpeaceenginelabs]

Ok, very honest. thx

For understanding: Does P2PT already make use of the gossip protocol? Or is Easypeers in the future even better, because of the gossip protocol?

[draeder]

I don't have much experience with P2PT really, so I'm not sure. It may connect all the peers together directly. Bugout certainly has a gossip protocol.

[worldpeaceenginelabs]

ok. But does Trystero something different than Easypeers/p2pt does then? Or are they both/all three doing the same? (swarm surrounding a "topic", by searching for a dedicated torrent?)

[worldpeaceenginelabs]

Because i like to integrate p2pt into Trystero, except i am thinking wrong, and they both doing the exact same thing... ???

[worldpeaceenginelabs]

Because if not, i like p2pt to be the fallback option of Trystero.

Every client is also a tracker itself, so its able to fetch connection adresses (DHT)

1. direct (webrtc) and indirect (torrents, IPFS) (webrtc prioritized, Trystero reconnects through torrents and ipfs as a fallback)
2. indirect/fallback (torrent/magnetlink) (P2PT) reconnect, than back to 1. (direct)

[draeder]

I believe Trystero/P2PT/Bugout/Easypeers all operate the same way when it comes to webtorrent.

The problem you will face with

Every client is also a tracker itself

Is that browsers cannot be trackers, because they are unable to listen for incoming network traffic without first making an outbound connection (e.g. to a tracker).

[worldpeaceenginelabs]

Is that browsers cannot be trackers, because they are unable to listen for incoming network traffic without first making an outbound connection (e.g. to a tracker).

The moment, the client is connected through WebRTC to other clients, exchanging DHT lists with them, isnt it a tracker then?

[worldpeaceenginelabs]

Imagine a client A already online 2h, has around 100 entrypoints on a dht. A new client B, just got online, has maybe 2-3 entrypoints on its dht. When client A and B connect via WebRTC, client B synchronizes with client A to get the other 100 entrypoints over WebRTC.

So client A is virtually the tracker for client B, or not?

[draeder]

Browsers can't use the DHT because they can't listen on ports.

[worldpeaceenginelabs]

Yes, i understand.

But two clients exchanging DHT's via WebRTC, is the same what trackers do, or not? Of course, you need a classic tracker for their first kiss, but then...

Where is my error in my thinking?

[draeder]

Trackers crawl the DHT to find peers, accept the WebRTC signaling from those peers, and send it to the other peers.

No new WebRTC connections can be created without some kind of way to coordinate this WebRTC signaling exchange. For example, Alice and Bob are connected to each other through Tracker1. Now Carl wants to connect to Alice and Bob, it is impossible for Carl to exchange WebRTC signaling data with Alice and Bob unless it is sent through the Tracker1 server. You could have 1000 WebRTC connections, all of them will have had to coordinate the WebRTC signaling somehow. In WebTorrent, those are the tracker servers.

In Gun DB, those are the relay servers.

In Hyperswarm-web, those are websockets servers.

Take simple-peer for example. It simplifies WebRTC signal exchange but doesn't ship with any way to actually coordinate the WebRTC signaling. Some kind of mechanism to do that is required. I've done it with a websocket server I built for this purpose. It could be anything though, like my idea to use MongoDB Atlas change streams, for example.

The biggest challenge with decentralization in browsers is this requirement for some kind of centralized way to connect the the peers together.

If what you're building can solve this challenge by using multiple fallback mechanisms for WebRTC signaling exchange, that would be awesome. Gun's AXE protocol is striving to do this too. I've been working with @amark on developing tests to achieve it. When AXE is done, it could probably be used as one of the fallbacks in True-P2P

[worldpeaceenginelabs]

Well explained. That was a big update. I got it now. Thank you very much.

If what you're building can solve this challenge by using multiple fallback mechanisms for WebRTC signaling exchange, that would be awesome.

Thats the goal ;)

When AXE is done, it could probably be used as one of the fallbacks in True-P2P

definitly

The biggest challenge with decentralization in browsers is this requirement for some kind of centralized way to connect the the peers together.

well, i am thinking about a p2p "API"?/"DHT"? or easy said, a list with direct entrypoints running on the swarm network.

As soon, as a client is connected, it receives multiple entrypoints to be able to get reconnected. (independant from torrents/ipfs) At the same time, all clients got that lists updated, so in case of a disconnect, they could just reconnect via the already updated lists (saved on the client), minimum one entry on the list will work out, and synchronize again on reconnect.

In turn, a client has always a hundred options of direct webrtc reconnects from the list. Or websocket, webtorrent, ipfs (and the other 10 options i mention in readme) as fallback.

So i am running my own lists for direct connection, and use the open available lists (torrent/ipfs etc.) as fallback.

[worldpeaceenginelabs]

Trystero is pretty much doing everything from my bucket list. So i "just" need to add third partys now.

Take a look at this: https://github.com/dmotz/trystero/issues/16

Thank you so much for suggesting Trystero to me 🙏🏼🙏🏼🙏🏼

[draeder]

Interesting.. I looked at Trystero a while back; I don't remember why I moved away from it. Notably, I'm working on building a decentralization based secure 1:1 tunnel at the moment. It could potentially be used for trust between True-P2P servers.

[worldpeaceenginelabs]

sounds like p2p vpn?

[worldpeaceenginelabs]

To me, it looks like, there is everything available for true-p2p already on github. Someone just need to patch that stuff together, do some custom coding, some finetuning... done

[draeder]

sounds like p2p vpn?

https://github.com/draeder/timebeam

I have it live at a minimally viable level now..

[worldpeaceenginelabs]

sounds like this repo but with onetime password? https://github.com/mafintosh/hyperbeam

Is hyperswarm a signaling method free to use?

[draeder]

Yes, Timebeam is based on Hyperbeam. Also, yes, hyperswarm is the networking layer of hypercore.

Also check out hyperswarm-web. Hyperswarm works only in Node.js. Hyperswarm-web is an implementation of hyperswarm that works in the browser.

[worldpeaceenginelabs]

Wow. Great. Thx I even understand whats happening. The source code is well commented and documented.

One could add further proxies and fallbacks here. And then customize the proxy/fallback handling of the repo with if, then, timeout conditions.

For understanding: Does hyperswarm depend on something? like if hyperswarm doesnt exist anymore, is their api still usable? So, future safe?

[worldpeaceenginelabs]

one could add webtorrent, ipfs and webrtc fallbacks by adding the trystero strategies from this folder https://github.com/dmotz/trystero/tree/main/src into https://github.com/RangerMauve/hyperswarm-web

just by calling it as a function after a specified timeout, if "const swarm = hyperswarm" times out.

[draeder]

All decentralization technologies out there require some kind of bootstrap peers. In the case of hyperswarm, there are 'root' bootstrap servers. In the case of hyperswarm-web, it's a websockets server you have to run yourself. In the case of webtorrent, it's the tracker server(s). In the case of gun, it's the gun relay servers. Until browsers allow listening on ports, this will always be the case.

I'm basically building a type of Trystero with Mishmesh. The main difference being that, with Trystero the connections are independent of the p2p network a peer is connected on. With Mishmesh, the peer connections will relay messages to other p2p networks regardless of which p2p network a peer is connected on. And, Mishmesh will allow adapters to be written to enable data relaying between any network.

Ultimately, True-P2P could use Mishmesh as a way to connect a peer on any decentralized network to another peer on any other decentralized network. The biggest challenge there is defining some kind of scalable gossip protocol (or daisy chaining [in Gun terminology] for messages). But that's what MMST could be used for, which is what hyperswarm-web uses and was the inspiration for Easypeers' gossip protocol.

I really love the idea of True-P2P. I can see Mishmesh as its transport layer.

[draeder]

I really love the idea of True-P2P. I can see Mishmesh as its transport layer.

The difference being, though, Mishmesh uses other p2p networks as the bootstrap to connect peers together, instead of as fallbacks.

Think about it like this:

1. Instead of p2p network 'fallbacks', Mishmesh uses p2p networks as the bootstrap into the Mishmesh network

2. After exchanging SDP over Mishmesh and some kind of True-P2P partial mesh connection logic and gossip protocol, peers are all connected together and can communicate in a scalable way

3. The peer connections using other networks are then dropped

4. The peers are now communicating over True-P2P

But, if Gun's AXE protocol is successful, Gun will be the main transport for Mishmesh.

[draeder]

peer <--> Gun AXE <--> (Mishmesh adapter) <--> Other p2p networks <--> peer

[worldpeaceenginelabs]

1. Mishmesh uses p2p networks as the bootstrap into the Mishmesh network

is the Mishmesh adapter in the topology above/mishmesh network, based on Bugout?

[worldpeaceenginelabs]

I really love the idea of True-P2P. I can see Mishmesh as its transport layer.

i am happy to hear and appreciate it

[draeder]

is the Mishmesh adapter in the topology above/mishmesh network, based on Bugout?

I'm trying to make it universal. E.g. as long as a developer using Mishmesh provides the adapters, it will relay messages between all of the adapters the developer provides.

I have it almost working between Gun and Bugout... I'll probably have it working this week. Then I can write up the Mishmesh API documentation, and from there developers can start using it between whatever p2p networks they want!

[draeder]

For example, bugout and gun are independent adapters to Mishmesh.

Mishmesh by itself won't do anything without adapters.

Although, I guess, I could provide default adapters... I'm not sure if I like that idea for this library, because it seems to be more True-P2p level stuff.

[worldpeaceenginelabs]

Ok, i imagine mishmesh as a armless kraken (client), but which you can add arms to (adapter) which adds network compatibility?

[worldpeaceenginelabs]

are you german? i just recognized, mishmash sounds like german Mischmasch :D

[draeder]

LOL -- German by heritage, American by birth. My last name means 'wheel maker' in German, derived from the word rädermaker. Mishmesh comes from the word mishmash, but instead of mash is mesh to describe that is is a mesh of mismatched things 🔥. I would venture, that mish has German roots.

[worldpeaceenginelabs]

interesting, i googled mishmash and it turns out, it means "A collection containing a variety of miscellaneous things." That is exactly what german Mischmasch means. 😅

[draeder]

interesting, i googled mishmash and it turns out, it means "A collection containing a variety of miscellaneous things." That is exactly what german Mischmasch means.

My latin is better than I thought.

[draeder]

Ok, i imagine mishmesh as a armless kraken (client), but which you can add arms to (adapter) which adds network compatibility?

This is exactly what I'm imagining for Mishmesh. But the kraken is a new network...

[worldpeaceenginelabs]

This is how i understand Mishmesh. But thats not different from what i imagined from true-p2p at the end. Your solution is just cleaner and simpler. Less susceptible for errors, because of smaller, simpler code.

A very high potential for adaption in the open-source, decentralized and jamstack community, because everybody is able to combine what they like.

You could add further adapters for famous libraries from time to time to raise the number of available networks and sowith the attractiveness of using Mishmesh. And i am sure, there would be fast a list of devs joining adapters for Mishmesh.

Because Mishmesh, as i understand it now, can do the scifi p2p from true-p2p, but is simple like Zapier