Hyperswarm network reliability and fallback

[urbien]

Problem

See the issue for this. Summary:

1. No UDP in browsers.

2. Corporate firewalls may block UDP. Need to bridge to DHT over WebSockets or WebRTC.

3. No peer discovery on Cell Phone networks. Cellphone networks run symmetric firewalls. So mobile apps or PCs on HotSpot can't establish direct connections (although UDP works, NAT hole punching does not).

4. DHT state needs stability. Peers that come and go lose DHT state and need to recreate it (although this can be overcome with caching). Peers that change their IP address too often, destabilize DHT.

This solution uses 2 servers for signaling.

Note that WebTorrent works in the browser. Need to investigate how they do it. They have been discussing "DHT over WebRTC" for a long time.

See a number of issues still pending resolution to make Hyperswarm and Hypercore work in react-native

Fallback using Personal Cloud

0. If in browser, fallback to 2.
1. check holepunchable indicatior in Hyperswarm. If false, fall back to next ..
2. get DNS record for bootstrap servers (multiple IPs). Tradle will dynamically populate DNS records using DNS server of the Data Center(s) in runs in.
3. choose IP randomly (or rely on a DNS client to give one) and request DHT lookup for desired my Personal Cloud (e.g. hash of Identity).
4. Mutual Authentication that this is our Personal Cloud. If failed, choose another IP.
5. Cache IP. Response needs a TTL, e.g. 24 hours.
6. Mutual Authentication that this is our Personal Cloud. If failed, repeat 2.
7. Forward all Hyperswarm request to that IP.
8. Read a feed from Personal Cloud with IP addresses of bootstrap servers. This avoid DNS altogether.
9. Next time IP does not respond or does not Authenticate, repeat 3.

Notes

• avoids the need for DNS records for Personal Cloud servers
• avoids the need for SSL certificates for Personal Cloud servers
• still has a choke point as relies on supernodes for the initial DHT lookup. But regular Hyperswarm also has bootstrap servers, so not worse.

Future research

To increase privacy, especially for applications like COVID-19 Contact Tracing, we need to add encryption to DHT traffic.

• To encrypt DHT traffic we need to add Noise, which adds several roundtrips (RTT). QUIC provides 0-RTT, but can it be used for hole-punching? I think it will as the principle is the same: 1) UDP traffic leaving your peer passed NAT and leaves it open for traffic on this port. 2) exposes NAT's public IP,Port to DHT server socket.
• QUIC alternative is Noise, which Hypercore already uses, and Noise offers 0-RTT. One of the Noise patterns, IK (which is 0-RTT) received formal verification from the WireGuard VPN authors.
• With 0-RTT you need to be especially careful with replay attacks. So doing updates to DHT on 0-RTT is not OK, but doing queries is safer
• QUIC creates dependency on TLS certificates, which Noise does not have
• QUIC / HTTP/3 will expose API in the browser to utilize UDP. This may allow DHT over QUIC. QUIC resolves the absence of UDP in the browser but still has an issue of DHT stability, which Personal Cloud overcomes.
• Cloudflare open-sourced QUIC / HTTP/3 implementation in Rust (so it may be able to run in WebAssembly in Node and browser).
• Chrome-specific (for now) RTCQuicTransport is WebRTC on QUIC with better RTT, and also using UDP and falling back to TCP. Will require same signaling as WebRTC.
• Geo-routing can be added to DHT

[balupton]

While I am a newb when it comes to low-level p2p, I imagine that the cloudflare workers stack (workers kv for persistent, durable objects for ephemeral) could be used to facilitate a peer discovery service, and multiple workers from different accounts could sync data to facilitate a multi-owner DHT (akin to bittorrent trackers, or scuttlebutt pubs), which could use some form of cryptography to verify what the worker says who is who is actually who is who. Such operating on the edge with cloudflare power would solve a lot of the current performance issues with DHT technology. So clients could specify several of these worker locations to replace the oldschool dht, or supplement it, and update their listing as time goes on.

seems zeronet are somewhat doing this: https://github.com/HelloZeroNet/ZeroNet/issues/1913#issuecomment-466756674