staheri14 commented 2 years ago

Background

The Status app is gradually and actively migrating to waku v2. This means the scalability and robustness of Status services will be directly impacted by the waku v2 protocols. One of such Status services is the Status community which relies on the Waku2-store protocol to serve historical messages. The present issue summarizes the constraints and requirements related to the Status community history serving that are identified by the Vac team during the last offsite and after multiple conversations with the Status desktop team. The objective of this issue is:

To summarize and fixate the identified constraints and assumptions
To synchronize the view of the two teams i.e., Vac and Status desktop regarding the current and next immediate state of 30-days history serving for the Status communities
To provide the Vac team with the initial problem definition based on which they will set out a plan to conduct relevant research to tackle each of these requirements in a timely manner.

Please read the issue, and share your viewpoints, especially regarding the next immediate state of the problem.

Status community 30-days history serving

Current state

Currently, the 30-days history of a Status community is served as follows. (Note that the 30-days history serving solution is orthogonal to the archival history serving for which BitTorrent is utilized.) The community owner runs a node(s) which is(are) supposed to be 24/7 online and persists message history for the last 30 days. In case the community owner goes offline, it will get help from a few other nodes i.e., Status/Waku fleet nodes (running waku2-store protocol/Mail servers in waku v1), to fill the gap in its history. All the community members can query the 30-days old historical messages from the community owner (or the Status fleet nodes).

The current design yields a reliable and consistent 30-days message history based on the following assumptions: 1- The underlying transport layer i.e., waku2-relay is reliable and robust. That is, messages will be reliably and consistently propagated in the network and will reach all the connected nodes unless the node itself experiences a connection failure or crashes.
2- The community owner and a few other known nodes i.e., fleet nodes (that run waku2-store protocol/Mail servers in waku v1) are reliable and available i.e., no crash failure and no Byzantine behavior (they act honestly).

Next immediate state

Based on the conversation with the Status desktop team, the first assumption i.e., the reliability of the transport layer seems to be a valid assumption given that no unreliability e.g., missing messages have been observed so far. Thus, it is safe to continue based on this assumption.

The second assumption is a strong and limiting assumption and needs to be relaxed for the following reasons:

It is not realistic to assume a highly available community owner
It is perceivable that due to the desire for more decentralization and privacy, we want to move away from relying on the Status fleet nodes
Having a central message history provider will not scale with the growing number of community members, and the community owner can become a single point of failure
The other nodes e.g., community members may be willing to contribute their storage to serve community history

The relaxed version of the second assumption would be:

The community owner is not the only node responsible for the 30-days historical messages
An arbitrary number of nodes may contribute to the 30-days history serving by running the waku2-store protocol
Nodes can have varying availability/online windows
The aggregate of the online windows of all the nodes that serve the 30-days community history spans the last 30 days
Nodes running the waku2-store protocol are honest and won't lie about their message history

Solution acceptance Criteria

The waku2-store protocol needs to be updated or possibly coupled with another protocol to accommodate an acceptable solution for serving the 30-days history of Status communities. The final design should work under the said relaxed assumptions and provide

[ ] Availability: other non-store nodes (those that do not run the waku2-store protocol) must have a way to fetch historical messages from store nodes for any period that falls into the last 30 days.
[ ] Consistency: and, that the query result should be consistent between every two such querying nodes.

Questions to the desktop team

Some aspects were not discussed during the offsite, so sharing them here:

What is the upper bound on the number of members in each Status community?
Is it a viable assumption to assume that only members of a community would be willing to serve community history? this might affect the final solution.

relevant links

Waku2-store https://rfc.vac.dev/spec/13/
Waku2-Relay https://rfc.vac.dev/spec/11/

staheri14 commented 2 years ago

cc: @oskarth @jm-clius please let me know if you have other points or questions to add to the issue. I will soon share it with the desktop team.

jm-clius commented 2 years ago

Thanks, @staheri14! Not sure if you want to include here, but this seems to be dependent on at least two other ongoing efforts: (1) The ability to store 30 days of history by utilising a SQLite-only store: https://github.com/status-im/nwaku/issues/950 (2) Some capability discovery/exchange mechanism to communicate online/offline windows for store nodes (mentioned in many issues, but original capture: https://github.com/vacp2p/rfc/issues/429)

staheri14 commented 1 year ago

@iurimatias please let us know your thoughts on this? especially on the assumption as well as timeline-wise.

jimstir commented 3 weeks ago

Issue moved here

vacp2p / research

Waku2-Store protocol: accommodating Status community history serving constraints and requirements #103