Satellite-im / Warp

Interface Driven Distributed Data Service
MIT License
14 stars 3 forks source link

Refactor rg-ipfs queue #141

Open dariusc93 opened 1 year ago

dariusc93 commented 1 year ago

The current queue is setup as a map which would check for peers connection status and send messages accordingly if they are online, however this would create a bottleneck with large amounts of messages and events stored in queue and would not be scalable since we check one peer at a time.

The queue should be refactored to be its own struct that would handle each peer as a entry with a separate task performing checks on each entry. When we wish to add something to queue for that specific peer, we would use a function to the queue that would send the data to the entry, If there one available. If there is no entry for said peer then one would be created. Inside each entry would contain a separate tokio task that would run and would perform a check on the peer connectivity. Once the peer is confirmed to be online, we would deliver all the entries supplied. After everything was delivered, we would signal that the queue entry is empty and the queue, which would also have its own task, would check each entry for its status and after seeing any items marked empty would begin a timer which would mark the removal the entry at the end of the timer.

Each entry would be saved in its own file that would be load into its own entry points and begin the task after loading it into memory.

Notes:

dariusc93 commented 1 year ago

With https://github.com/Satellite-im/Warp/pull/217 came changes that introduce a request/response style system. This system could possibly be used to replace the queue itself by requesting messages from since a point to other users in the chat, in which the user would respond with any messages, however this comes with a problem in detecting deleted or edited messages when making a request.

Possible solution in the event we decide to go this route: 1, Create a new internal message that is ignored by the interface but is used to reference specific events such as pinning, unpinning, editing and deleting and pass those along to point to messages that were changed in some way.

  1. Queue those specific events instead and request the queue from the user for those events prior to requesting any messages.