jgowdy
commented
3 years ago We will have to look at what kind of fast compression we want to apply to the message payloads before we encrypt them.
There are some interesting numbers here: https://www.lucidchart.com/techblog/2019/12/06/json-compression-alternative-binary-formats-and-compression-methods/
We will have to look at Brotli, Zstd, gzip -0, xz, etc and see what provides the most benefit for our use case with the least impact. Networks are usually fast anymore, so you don't want to spend too much time compressing. But some light compression could take a bite out of the encrypt / decrypt / signing size.
Nodes should be configured with a hostname upon deployment which is resolved to a set of AAAA or A records which provide public facing ports to join the mesh.
Once nodes join the mesh, they can use host discovery to find the other hosts.
When bootstrapping, a node creates a public/private key pair. The public key becomes the node's identity.
In the interim, we should configure clients with a static membership list until we have message passing working.
Nodes can be configured to load different modules, and those modules can provide different capabilities. For example, a node configured with an EmailSend module with a configured SMTP / Submission endpoint would advertise the SendEmail capability.
Nodes will continuously attempt to make one or more outbound connections to the other peers that the node is aware of, on the paths the other node is advertising, until the configured number of outbound paths are up. This happens regardless of any inbound paths from the same host.
When a Node connects to an endpoint, both the originating Node and the terminating Node will immediately send announce messages describing their identity, capabilities, etc. Nodes will keep a list of active paths / connections to each other Node, and when a packet is to be sent the connection to be used is prioritized based on the different preference levels of protocol, layer 2 or 3 adjacency, encrypted transport, and other relevant traits. Nodes will prefer to send messages on the outbound connections they make. This way packets are distributed among at least two of the links, keeping both alive.
Nodes can have multiple IP addresses and multiple ways they can be accessed. Our current focus is on WebSockets as a transport but we can add more later. But the WebSocket links can form across multiple IPs between the same Nodes. Either way, for each send() a single connection will be selected and used.
By default all traffic should be point to point. For hosts with public IP addresses, this is not a problem. For hosts behind NAT who have NAT-PMP or UPnP should be able to map a port and also accept inbound connections. However, some Nodes won't be reachable inbound. Those Node should still be able to make outbound connections to port 443 on other Nodes who can accept inbound connections. Other Nodes will be attempting to connect to the published paths of the Node that cannot accept inbound connections, but with a back off that will settle down to trying once every N hours over time. As long as one bidirectional WebSocket link is up, the Nodes can pass traffic.
Nodes send JSON messages to one another. The message JSON always starts with a map, and the root of the map contains source (public key), destination (public key or wildcard for broadcast), a TTL counter that defaults to 2 (see adjacent hop routing), and a signature for the payload that matches the source public key. The payload is one of the items in the root map, and it is a Base64 encoded encrypted JSON map containing the application specific payload values. The message JSON also has a Headers map under the root for providing metadata such as whether the payload has been compressed before encryption and what compression was used. The payload is encrypted using the destination Node's public key so that only it can decrypt the payload.
When receiving an inbound message, a Node first evaluates the source field. If the public key in the source field is trusted as a member of the mesh, continue, otherwise drop message. Then the signature is checked on the payload using the source public key, if it matches, continue, otherwise drop message. Now we know we have an authentic message from a trusted member of the mesh.
The destination field is evaluated. If the message is for us, we go down the process received message path.
If the message is for broadcast/wildcard, decrement the TTL and then forward the packet to all connected Nodes over the most preferred path.
If the message is unicast, and we have a connected path to that Node, send it. If we don't have a connected path to the target Node, we don't try to make one while the message waits, we attempt to locate a Node in our "routing table" that does have a connected path to the destination node. We then forward our message to that node, which we then would hope would forward it on using this same logic. If we can't find any adjacent Nodes to send the message to, we can search our routing table / network map for anything up to N hops away from the destination node to forward the message to.
If we can't determine where we should send the message, we can optionally "scattershot" it to all of our connected Nodes over the most preferred path to each, in the hopes that the message can be forwarded by those Nodes.
Messages will include a field specifying the desired response type. "FireAndForget" messages are never responded to. "Acknowledge" messages are responded to with an ACK message once the packet is received and signature checked and decrypted (but before the message payload is processed). "RPC" messages expect an optional ACK before packet processing and then a Response packet with the JSON result payload for the operation / transaction. Any message type other than FireAndForget, a Node receiving a unicast message destined for a public key / Node that this Node has no knowledge of or rational path to, sends a NoPathToNode response.
Once Nodes form best effort meshes of WebSockets and are able to transmit messages to one another, we add a Raft implementation with the communication tunneled over our WebSocket messaging system rather than via direct network traffic. The Raft implementation will have an election. We will bias the election towards nodes with public IPs, more powerful CPUs, lower latency to test endpoints, longest uptime, and other factors. If no public IP hosts are candidates, RFC addressed Nodes which have seemingly functional UPnP or NAT-PMP are next preference. These hosts while behind NAT are able to accept incoming connections. Nodes that are behind NAT with no mapping protocol will always choose to be followers and never put themselves up as candidates.
Even if every other Node is behind NAT with no port mapping option, as long as there is one Node that has UPnP or NAT-PMP the mesh can form.
Nodes will send announcements on their connected paths periodically, and immediately upon initial connection. These announcements will contain the advertised paths for the Node, it's public key, it's capabilities, etc. These announcements will optionally contain the list of other Nodes / paths that the announcing node is aware of, feeding Node and path discovery.
Once the WebSocket mesh is up and we have a working Raft election, we now have a distributed peer to peer messaging system, but with an elected leader. The protocol then supports a key-value-store form of persistence. Each Node will have a replica of the information in the key-value-store. The Nodes can perform weak reads by reading their local values. The Nodes can perform strong reads by requesting a read from the leader. All mutations must be sent to the leader. The leader then replicates the mutation to all nodes (including the node that made the mutation, since it doesn't exist until it's replicated to us). Reads will include a modified time for the values returned, and Conditional Updates with that modified time will be checked as a form of optimistic locking. In effect, if the read that a write is based on is outdated, the write is rejected and the Node CAN choose to re-read and mutate again based on the updated record.
Nodes that join the mesh and are verified are able to queue replicated writes, download the current persistence database from any other connected node, and then apply the queued replicated writes until up to date and resynchronized.
The other persistence that could be offered would be a savvy form of object storage using erasure codes and distributed data across the nodes, or a much simpler file system based distributed file store with configured number of replicas.
Once the leader election takes place and persistence comes up, the leader can undertake to map the network using broadcast and TTLs, and render a network map into persistence.
Nodes would constantly be learning about peers, but only registered peers in the master peer list are considered valid/trusted. Once a Node bootstraps and joins the mesh, it is added to the master peer list in key-value-storage, and then it downloads that master peer list.
Nodes would potentially make untrusted adhoc connections to discovered paths, but when they receive the announcement after connecting they'll learn this discovered path leads to a node not on our list.
Only if the other Node is part of the master peer list, do we handle or process traffic from that Node.
Master peer lists are only additive to peer host discovery, therefore if strong reads aren't desired or available, Nodes can leverage their own local copy since the last replica update.
At the moment paths will be WebSockets, which have 64bit size_t message size limits. However, regardless of how high they are, any path will advertise what it believes to be its maximum message size (we may probe this). Paths / transports that can't handle the size of the message we are sending are simply disqualified/ skipped.
The Raft elected leader would be able to act as a distributed lock manager between nodes, allowing for synchronization. With the key-value-store, we have persistence. With multiple paths we have multi-homing.
With the distributed lock manager and key value store provided by the leader, we can then leverage the JSON messaging mesh to bootstrap a WireGuard VPN mesh, similar to Tailscale. Effectively the JSON messaging mesh becomes the D channel and the WireGuard links become the Bearer channels.
Down the road if hosts can't setup WireGuard links due to firewalls or NAT issues, we can have bearer/IP channel fall back to an OpenVPN outbound TCP 443 which should work on any host that can make HTTPS requests.
Once the IP mesh is up, the JSON messaging mesh should advertise a new path, over that IP mesh. That path would likely be preferred. This would allow the control plane to pass through the VPN security of the data plane in nominal IP mesh up conditions, falling back to direct WebSocket linkage. Inception. In theory once the IP mesh is up, all non-IP mesh control plane connections could fall away, leaving only the IP mesh as active paths. These paths would be used until there were connectivity issues that caused the Node to shift back to trying to do direct WebSockets.
Once you have a self forming JSON messaging mesh that provides distributed persistence with optimistic locking for consistency, Raft for resiliency and consistency, and distributed lock management, which then bootstraps you a peer to peer IP VPN mesh, you basically have a powerful platform for many applications.
One, a personal botnet for nmap scans, trace routes, personal VPN service, monitoring my servers and desktops etc.
Two, a family could mesh their computers together and appear to be on the same LAN allowing for file and printer sharing for example.
Three, certain multiplayer games may be playable as LAN games through the WireGuard mesh.
Four, a small business could, rather than building an office network with the complexity that comes with that, use this system to build a zero trust network between all of the office employee workstations and the office servers. This would allow the office LANs to be configured the same as a Starbucks. Standard Internet with a standard NAT router. The Office LAN would be treated like a very fast, very optimal public WiFi, rather than a trusted network. The system would offer different modules that allow for health monitoring of the Node for example. Disk space sufficient? OS updates applied? OS isn't EOL? Antivirus is enabled? Etc.
Five, rather than setting up a distributed IP VPN mesh, the control plane could be used to create a virtual/orchestrated firewall which would act like a parameter firewall, but would accomplish the rule set by translating those rules into host based firewall rules.
When a Node is on an RFC address, it advertises that path too. When a Node is attempting to connect to another Node on one of its advertised paths, if the source Node has an RFC address, it will try to connect to the other Node's RFC addresses. This allows Nodes on an Office LAN to pass traffic between hosts on the local switch.
When you install a Node, configure it with the bootstrap hostname, it tells you it’s public key as it starts up. The Node will try to connect repeatedly but the peers won't trust it because it's not on the master list. The person managing the node add issues a command to one of the trusted nodes, doing a mutation to the master peer list to add the public key of the new Node. At this point the new Node's repeated efforts to link to the mesh will start working (hopefully).
Using the key-value-store functionality or optionally transparently plugging in a third party key value store, the mesh could offer application layer key value store eventually consistent persistence with optimistic locking. This is effectively the "DynamoDB".
Using the object storage functionality previously discussed, or plugging in a third party object store, the mesh effectively provides "S3".
Endpoints can be configured to run scripts based on events such as a particular URL path. These scripts would be written in something like JavaScript, Python, Lua, etc. The scripts would be able to run other scripts, potentially modify messages before send or after receive, etc. The scripts would be able to access the key-value-storage and the object storage and send messages. Any time we need user logic or want to offer the ability to customize logic. We would likely pick one language to really put most of our effort into. This would provide "Lambda".
The zero trust IP networking through a bootstrapped WireGuard VPN mesh (with OpenVPN/TCP/443 leaf nodes as needed) provides what is effectively the "VPC".
At this point we have the makings of a zero trust private mesh network that would work across the world, with a built in private cloud.
When we do start getting working builds going, I would like us to make sure the meshd engine announces itself in syslog, in Windows event viewer, in the Windows System tray or notifications area, as a MacOS notification, whatever. The point is, our copy of the command and control code makes every effort to make it clear that we are installed and running and to not be "hidden" on someone's computer. Most people think "Botnet! Evil!" We can't stop people from taking this open source software and recompiling it without the alerting code. But perhaps by making sure our software is very plainly open about being installed and running, we can avoid malware engines deciding we are a botnet.
We might also make it super easy to rip out, like one click removal. I just think we need to be very proactive about this from the start or we could get malware flagged and it'll be a nightmare trying to deal with that.