jacobheun commented 5 years ago

I've started thinking more about Connection Management for the v2 spec. Adding my thoughts here, although they are still very raw and incomplete. The general approach is to leverage allocable overlays/Topologies to create a proactive approach to Connection Management, instead of the current reactive approach.

Connection Management

Topologies

Multiple systems that integrate/leverage libp2p create different Network Topologies. In most instances, these are generated by the natural usage of those systems, for example the DHT and Gossipsub. In systems like IPFS, it is desirable for nodes to create custom Topologies to better support certain network needs. For example, JS IPFS nodes running in the browser may have a high dependency on Preload nodes to store their content in instances where they cannot stay online.

Bootstrap nodes are another instance of a network Topology. While their connectivity may be short lived, they serve as a critical role in remote network discovery.

It is desirable for nodes to be able to persist these Topologies in the event they go offline. When the node comes back online, being able to rejoin their Topologies quickly, will enable users to resume their normal activities in a shorter period of time.

Topology Interface

Creating an interface for Topologies may enable us to support a broad range of implementations. Users would be able to define custom Topologies for their applications, such as High Priority nodes. Subsystems of Libp2p could leverage the interface to create and evolve their own Topologies. Topologies could be created to control their limits. For example, too many nodes attempting to add IPFS Gateway nodes to their Priority Topology could be denied, or directed to other high availability nodes in the Gateway nodes Sibling Topology.

Connection Manager

The current libp2p Connection Manager is reactive. When connection limits are exceeded, connection pruning takes place. While the Connection Manager attempts to take a best guess pass at pruning, it can currently cause starvation in subsystems if any one system is leveraging the majority of connections.

If the Topology system was leveraged, it could be made to request limits from the Connection Manager for each Topology. Each request could also define the types of connections being requested. For High Priority nodes these could be permanent connections, where as the DHT might request a mix of permanent connections, to maintain a consistent DHT overlay, and something akin to a burst set (many short lived connections).

Registration of Topologies with the Connection Manager would enable the Connection Manager to make more informed decisions when pruning, and in addition, would enable the Connection Manager to deny requests. As subsystems are made to dynamically improve over time, future iterations of this registration may include a "repurchase", where systems may request additional connections if they are being heavily used. In addition, the Connection Manager could potentially, proactively loan or give an allotment of connections if it sees the system being starved.

Benefits

User and subsystem defined/managed Topologies
Topology limits requested from the Connection Manager
Enable researches to define Topologies such as Genet
Give users greater control over peer sets ipfs/go-ipfs#6097

vasco-santos commented 5 years ago

Nice ideas leveraging the topology definition @jacobheun

I like the idea behind persisting the topology, as a node shouldn't need to start from 0, each time it starts.

vasco-santos commented 5 years ago

After discussing the pubsub subsystem refactor with @jacobheun we had some discussion in this context. Keeping them here for visibility and future consideration:

Considering an example of Gossipsub with support for Floodsub fallback, how would we prune connections if we need to?

Weighting the protocols could be an option for a topology. So, say you're maxed at 100 connections and you've weighted gossipsub higher. Connection Manager gets a new connection that has gossipsub support, it could notify you of that and then mark a floodsub only peer for disconnection.

libp2p / notes