search

dao-xyz / peerbit

P2P database framework with encryption, sharding and search
https://peerbit.org
Apache License 2.0
166 stars 11 forks source link

STUN server network for bootstrapping apps #134

Open marcus-pousette opened 1 year ago

marcus-pousette commented 1 year ago

If #133 works out well, then it would be interesting to see whether we could create a cheap network that can bootstrap apps and make the app-making ridiculously easy and fun. Peers could host these kinds of stun servers with low cost, or at home with port-forwarding, and their would be little or no bandwidth costs for participating in an app network. Thanks for pointing this out. @TheRook

TheRook commented 1 year ago

An easy approach to STUN bootstrapping is a unique ID and peer.js. So, lets say hash of the publickey (which is the 'wallet address' in cryptocurrency networks). Then use this unique ID to form a connection with that host using a predetermined STUN server and then form a connection (a peer.call() in peerjs: https://peerjs.com/). A client could have a list of lets say 10 common STUN servers and we all agree that the order of the servers is the same - so the network will fallback in the same cascading order (we hope).

One problem is that an adversary can connect to the server first and register the peer-id used in peer.call(), and this is just a downfall of STUN/TURN, so you still have to make sure the client is who they say they are. And if they aren't you could fallback to another STUN server or another unique ID on the same server until you get a valid handshake. To make matters worse, an attacker could just connect to every STUN/TURN server and register the same key and disrupt communications - but even in this case we should not assume an adversary can make an unlimited number of connections - Google's STUN server is hardened against DDoS.

That being said, there are benefits in having an HTTPS server help out with bootstrapping. For example - let say the client is given an HTTPS server, a list of STUN/TURN servers and then falls back to libp2p for discovery - which settles the argument of it being fully distributed. Now if lets say there is an HTTP-rpc a webrtc-rpc and a libp2p-rpc version - well in that case a dev could host a beefy supernode on Azure/GCP/Alibaba/Oracle cloud - they all offer free linux nodes, and they also have a free-tier serverless+cloud database - all of which could be used together.

Now if a new node comes in and wants to sync the full DB - ideally there would be some snapshot of the DB that could be distributed via https://github.com/webtorrent/webtorrent - where each individual CRDT has been settled by a trusted party - and this settled snapshot is signed and distributed. Basically when a full node connects to Ethereum every block needs to be fetched over libp2p - well a similar solution is needed for this DB.

TheRook commented 1 year ago

Another approach that maybe less error prone is that HTTPS is always used to to lookup where to connect. A collection of peerbit nodes can elect hosts to wait for new incoming connections on a chosen STUN server. The HTTPS RPC determines the current meeting ID and STUN host for the service they are connecting to.

You may need to distribute a list of possible connections and then a new client simply goes down the list in order looking for an available connection. A node that is under served can perhaps sit around with few open connections pending. Although having pre-fork'ed connections ready would reduce connection times, it could be easily flooded.

If a client fails to connect - there could be a mechanism whereby the hub reaches out to a group and requests that a new STUN connection be formed for a new node joining the network.

Just a thought.

TheRook commented 1 year ago

So, I believe we could effectively tunnel SDP 'Session Description Protocol' over any other protocol or side channel (HTTPS/WebRTC, Hardcoded or QR Code or URL or DNS-SRV record...) https://github.com/clux/sdp-transform

These SDP records can be built by a node that is aware of network health, and typically contain a list of ICE candidates for a specific group: https://developer.mozilla.org/en-US/docs/Web/API/RTCSessionDescription/sdp

For example, if the billing API is reporting that your freetier usage is getting full - then you can prioritize other nodes for establishing new connections. Baring 'running out' of any resources, a good first approach is a basic round-robin load balancer, and then at some point you could introduce weighted round robin load balancing (https://constellix.com/news/load-balancing-round-robin-vs-weighted-round-robin).

Fetching long-lived SRV records via DNS-Over-HTTPS gives you a free distributed K/V cache. Also, fetching over HTTPS helps with clients helps preserve privacy, and works well bypassing some censorship like what is found in Indonesia, but not the GFW. The benefit here is that a single node could act as a DNS for a massive number of hosts - it is known to scale well.

marcus-pousette commented 1 year ago

An easy approach to STUN bootstrapping is a unique ID and peer.js. So, lets say hash of the publickey (which is the 'wallet address' in cryptocurrency networks). Then use this unique ID to form a connection with that host using a predetermined STUN server and then form a connection (a peer.call() in peerjs: https://peerjs.com/). A client could have a list of lets say 10 common STUN servers and we all agree that the order of the servers is the same - so the network will fallback in the same cascading order (we hope).

You kind of run into this meta problem where you want this bootstrap configuration that is a list, but it also needs to be modifiable and owned by the community. Also you want to associate it with all kind of live metadata, such as load, free-space etc. This meta db can also be a Peerbit DB, but then you have not really solved the problem with bootstrapping, because this DB also needs to be discovered and shared. (1)

One problem is that an adversary can connect to the server first and register the peer-id used in peer.call(), and this is just a downfall of STUN/TURN, so you still have to make sure the client is who they say they are. And if they aren't you could fallback to another STUN server or another unique ID on the same server until you get a valid handshake. To make matters worse, an attacker could just connect to every STUN/TURN server and register the same key and disrupt communications - but even in this case we should not assume an adversary can make an unlimited number of connections - Google's STUN server is hardened against DDoS.

Peerbit is using libp2p and the STUN server equivalent here is a what is called a c "circuit relay server" which kind of solves the same problem as with STUN server. In this system you can but constraints on the connection handler to mitigate risks with DDos. The benifit here is that the relayed connectino gets an associated ed25519 publickey that you can use to sign messages with and what not, so you can use the same id for the handshake as for the messages you send through the connection, like your chat messages

That being said, there are benefits in having an HTTPS server help out with bootstrapping. For example - let say the client is given an HTTPS server, a list of STUN/TURN servers and then falls back to libp2p for discovery - which settles the argument of it being fully distributed. Now if lets say there is an HTTP-rpc a webrtc-rpc and a libp2p-rpc version - well in that case a dev could host a beefy supernode on Azure/GCP/Alibaba/Oracle cloud - they all offer free linux nodes, and they also have a free-tier serverless+cloud database - all of which could be used together.

Yeap, well right now you can very easily setup a node in a server center with a domain to get SSL working, 3 commands. It will create a libp2p node that can help with bootstrapping and can run on free-tier instances.

Now if a new node comes in and wants to sync the full DB - ideally there would be some snapshot of the DB that could be distributed via https://github.com/webtorrent/webtorrent - where each individual CRDT has been settled by a trusted party - and this settled snapshot is signed and distributed. Basically when a full node connects to Ethereum every block needs to be fetched over libp2p - well a similar solution is needed for this DB.

This already exists. You can invoke iteration request on a db and put "sync: true" flag on the Document store to collect all states https://www.peerbit.org/modules/program/document-store/?id=syncing

Another approach that maybe less error prone is that HTTPS is always used to to lookup where to connect. A collection of peerbit nodes can elect hosts to wait for new incoming connections on a chosen STUN server. The HTTPS RPC determines the current meeting ID and STUN host for the service they are connecting to.

You may need to distribute a list of possible connections and then a new client simply goes down the list in order looking for an available connection. A node that is under served can perhaps sit around with few open connections pending. Although having pre-fork'ed connections ready would reduce connection times, it could be easily flooded.

If a client fails to connect - there could be a mechanism whereby the hub reaches out to a group and requests that a new STUN connection be formed for a new node joining the network.

Just a thought.

See (1).

Also one has to put into consideration that many apps requires you to be able to fetch data immediately on load, so there can not be that much looking/waiting if you want to be competitive with Web2 solution. There is a tradeoff of creating a robust solution vs a fast and fragile one. The best solution are when all the reasonable assumptions has been identified and taken advantage of, in which we do short-cuts in the algorithms.

E.g. Peerbit sharding uses the hash of the commit to choose replicators automatically without any communication needed, because we can assume that peers will not change their ids to often.

For bootstrapping networks, or, build connectivity across apps however we need to think about what optimization problem we are solving and see if there is a nice solution that can be made by making good approximations of user behaviour. Just an annoying edge case is where you want participate in two apps/dbs and these are part of "different" clusters of nodes. Imagine now you can only make 8 connection, to which nodes would you connect? Now you have to think about more that just having a backup plan if a connection fails, but different apps might have different throughput in data, some requires low latency, some requires a secure connection but not low latency. Some apps might have more short lived connections than other. Some apps are more important than others. If you can objectively quantify a objective function you want to minimize, you can work backwards from that to see what kind of redundancy solutions you need in place, and how you can automate things.

So, I believe we could effectively tunnel SDP 'Session Description Protocol' over any other protocol or side channel (HTTPS/WebRTC, Hardcoded or QR Code or URL or DNS-SRV record...) https://github.com/clux/sdp-transform

These SDP records can be built by a node that is aware of network health, and typically contain a list of ICE candidates for a specific group: https://developer.mozilla.org/en-US/docs/Web/API/RTCSessionDescription/sdp

Yeap! I thought about modifying https://github.com/libp2p/js-libp2p/blob/daeb43d8821d2df1999871797a22fbdf502731f8/packages/transport-webrtc/src/private-to-private/handler.ts#L92 at some point...

For example, if the billing API is reporting that your freetier usage is getting full - then you can prioritize other nodes for establishing new connections. Baring 'running out' of any resources, a good first approach is a basic round-robin load balancer, and then at some point you could introduce weighted round robin load balancing (https://constellix.com/news/load-balancing-round-robin-vs-weighted-round-robin).

Weighting is not implemented yet but soon: https://github.com/dao-xyz/peerbit/blob/b297c190dde46617a158e8bd5bb182ac5dbe71af/packages/programs/data/shared-log/src/role.ts#L19

Fetching long-lived SRV records via DNS-Over-HTTPS gives you a free distributed K/V cache. Also, fetching over HTTPS helps with clients helps preserve privacy, and works well bypassing some censorship like what is found in Indonesia, but not the GFW. The benefit here is that a single node could act as a DNS for a massive number of hosts - it is known to scale well.

Yes I mean, as we talked about before. Eventually the solution for bootstrapping in the end will converge to something similiar to DNS servers, because both more or less solve the same problem. Reinvent the same thing is not a good idea unless you are solving a problem with the existing solution. IMO the issue with the current setup is that it does not scale outside the traditional networks that relies on nodes to have a domain names with valid certificates. For example fetching from Mobile phone -> Bluetooth -> Laptop -> WebRTC -> Laptop -> Wifi -> Raspberry PI should just work. and the technical solution behind this should not differ from other connectivity setups to much.

TheRook commented 1 year ago

So this is more of a free form list of ideas, not all of them have to be used together. Although there are many ways to skin a cat, there is one way here to dramatically reduce overall connection time during bootstrapping.

This already exists. You can invoke iteration request on a db and put "sync: true" flag on the Document store to collect all states https://www.peerbit.org/modules/program/document-store/?id=syncing

The sync is one part of the solution here. The idea is that if you already have a beefy HTTP supernode - lets say a freetier cloud function - then you could make a sync backup and load that over HTTP before joining the group - which will cut down on connection time. A single cloud function using something like dynamo could be tasked with not only aiding in new connections to the swarm - but also making sure that the node is updated prior to joining.

Now, in web2 - clients are transient, so any trusted supernode could respond to the client's request - so the first RPC might not be a sync - but rather a basic query or a search request.

Imagine now you can only make 8 connection, to which nodes would you connect? Now you have to think about more that just having a backup plan if a connection fails, but different apps might have different throughput in data, some requires low latency, some requires a secure connection but not low latency.

So - we aren't in a vacuum here, and many of the hard questions here have a possible solution, and in the case of round-robin load balancing, there are a number of examples on github. Also here are some great papers specifically on the topic of self-organizing round robin load balancing:

https://link.springer.com/chapter/10.1007/978-3-642-20344-2_8
https://ieeexplore.ieee.org/abstract/document/4274893
https://ieeexplore.ieee.org/abstract/document/7034709

... But the number of hours in the day is short, and the best solution is the one that works today. Start with the simplest algorithm that can manage. In this case - lets assume we start off with a single multiplexer - this single node will onboard everyone into a single message bus. You can monitor health of this node by looking at latency, or how many resources you have available. Nodes in the swarm that are in good health (can accept incoming connections from VPN clients for example :-), then these can be elected as supernodes - and then the main hub can turn down connections - and then a list of supernode candidate peers from this hub can be selected at random to form new connections with any new party.

... don't assume the first version has to scale perfectly. A central messaging hub will make sure that each leaf has reasonably low latency - if you have 100 nodes - then the ICE candidates list could be a random selection - evenly balancing out the load between leafs - or if you know some leafs are more powerful then when you create the SDP list of ICE candidates you weight the random number generator to prefer handing out more powerful nodes. Leaf growth is N+1, unlocking new bandwidth capacity at the cost of increased latency... which by all accounts should be a lot lower than libp2p.

But yeah, start simple.