oskarth
commented
2 years ago To make it clear in a public comment: I think this is a great breakdown. What is missing is more high resolution engagement on this by the people working on Community, so we can come to a shared understanding of the problem space, prioritize the work and figure out most critical next steps to work on together, including more concrete design (spec, interface with other protocol) and work implementation (how it'll interact systems) work.
As far as I know, this largely falls under Desktop and @iurimatias since this is the most important product priority to them. If something has changed in terms of who will be driving this, feel free to say so here.
0x-r4bbit
staheri14
Problem
This issue is the continuation of a previous issue https://github.com/vacp2p/rfc/issues/420 In this issue, we shall take a modular perspective to break down the community history problem into smaller and approachable sub-problems and then discuss each in isolation in the respective issue.
The problem is broken down into smaller components illustrated in the following figure. As the result of the interaction of these components (as shown in the figure) the community history would be stored and served to the community members in a reliable and scalable manner.
Each component has a specific functionality (which is presented next) and is associated with a well-defined interface (to be defined).
Each module can be implemented by different existing/future protocols/tools as long as the implementation respects the defined interface and functionality. Some components are optional and are marked with a green plus sign.
We will use the terms consensus nodes and store nodes interchangeably.
Message Ordering Mechanism This is about providing canonical order for messages, e.g., through DAG, vector clocks, Lamport timestamps, or any other method. Supporting message ordering (in general) is considered optional (but for the community history it is going to be mandatory).
Access control management/ Group management This layer is to keep track of the current authorized members of the community and the channels together with their read and write access level.
The output of this layer (if present) is input to the consensus layer. Supporting access control management is optional. Access control can be managed in a centralized way e.g., via the community owner who maintains the list of current members with their signature verification keys
Store synchronization/Consensus In this layer, a set of consensus nodes i.e., storage nodes interact with each other to get to consensus on their views of the historical messages. As the result of this protocol, all the participants (i.e., store nodes) will have replicated message states, hence, each individual store/consensus node can reliably serve the message history to the community members.
Historical messages can be treated in smaller groups of messages called view. The grouping can be based on time or number. Nodes in the store synchronization/consensus protocol come to a consensus regarding
This protocol can be implemented in a centralized way or can involve more entities. For example in the central case, only the community owner provides the storage service and does not have to synchronize with any other store nodes.
Discovery Consensus nodes can use this layer to find each other
Transport This layer specifies how messages are transported to the participants of the consensus/synchronization layer e.g., via Libp2p GossipSub
Offline/long-term storage This layer is to provide long term storage for old historical messages through e.g., IPFS, Torrent
While interacting with this layer from the synchronization layer, we need to take of things like how messages are packaged up and sent to this layer, and how users can fetch the packages from this layer, etc.
Resource Throttling Consensus nodes must be able to throttle
All these configurations will affect the communication between the consensus/synchronization layer and the long-term storage layer. For example, the storage and bandwidth constraints will affect the frequency and the number of messages that should be offloaded to the long-term storage layer.