search

MatrixAI / Polykey

Polykey Core Library - Open Source Decentralized Secret Sharing System for Zero Trust Delegation of Authority
https://polykey.com
GNU General Public License v3.0
27 stars 4 forks source link

Network Layer (next gen) for Complete Connectivity - this represents a revolution in network connectivity! #713

Open CMCDragonkai opened 3 weeks ago

CMCDragonkai commented 3 weeks ago

This is a high-level issue, a large epic with multiple branches.

Requirements of this design

  1. Complete connectivity is to ensure that all Polykey nodes regardless of their network environment can connect to each other if there is a sufficient link (potential multistep) between them.
  2. The ability for 2 nodes in symmetric NAT to discover and connect to each other.
  3. The ability for all nodes to act as relays - and thus decentralized relaying without rely on the seed nodes.
  4. The ability to ensure network segregation, firstly through mainnet and testnet separation, and subsequently private networks in PKE.
  5. The ability to simulate the network - create a "chaosnet" where one can deploy autonomous Polykey agents to perform random tasks in a loop, in order to test network stability empirically.
  6. The ability to do decentralized signalling efficiently, through each hop, that is user acceptance testing and simulation testing must be done according to #365 that was closed.
  7. The ability to attempt network passthrough routers and other equipment via UPnP #536
  8. Optimised mdns for local network discovery - to avoid conflict with existing mdns stacks, and integration into existing OS stacks.
  9. The ability to signal AND relay over IPv4 and IPv6 and change between the 2.
  10. Increased bandwidth with optimised QUIC implementation by parallelising the send and recv calls.
  11. Get past corporate firewalls that block UDP, but enabling optional HTTP based relaying.
  12. Increased debug transparency and audit transparency - local to the node and global to the dashboard whether public or private.

Additional context

Specification

Network Identity and Membership

This is needed to separate mainnet, testnet and private networks, possibly allowing PK to connect to multiple networks simultaneously but forming segregation on every command. It's probably easier to just make PK connect to only a single network.

We also need to figure out the exact role and relationship between the certificate identity chain, and the sigchain claim chain.

Decentralized Signalling and Relaying Algorithm Design

The initial design is in #365. However it has not been applied to relaying yet, nor has it been empirically tested.

Relaying ultimately has to be done through network encapsulation. We have to make use of our RPC to pass through binary blocks from one node to another. This will need to make use of our RAW mode in our RPC for efficiency reasons. We may need to test the performance of this though, because QUIC does actually have a datagram mode, I'm not sure if this is relevant here.

And DOS prevention in https://github.com/MatrixAI/Polykey/issues/556.

Network Simulation/Visualisation

Involves the usage of Orchestrator and Pulumi.

Chaosnet

It will be important to create autonomous looped Polykey agents to be able to simulate a dirty and chaotic network to see how robust our mechanisms are.

NodeGraph Garbage Collection

The NodeGraph since #537 is capable of multi-host addressing. However GC of the NodeGraph has been neglected, and I believe fast manipulation of this node graph will need to be dealt with.

Discovery Performance and Efficiency

MDNS and related algorithms suffer from some in-efficiencies.

QUIC Performance

https://github.com/MatrixAI/js-quic/issues/54

IPv4 and IPv6 Conformance

https://github.com/MatrixAI/Polykey/issues/669

Sub-Issues & Sub-PRs created

  1. ...
  2. ...
  3. ...
linear[bot] commented 3 weeks ago

ENG-305 Network Layer for Complete Connectivity

CMCDragonkai commented 3 weeks ago

I expect to promote this issue to a "Project" as it something that would fit onto a roadmap. As per the discussion on the Project Graph concept - pipeline parallelism would make sense down a particular Project Path, so we can incrementally delivery "Project" rather than spreading the team across multiple "Project" each with their own paths.