AppFlowy-IO / AppFlowy-Cloud

AppFlowy is an open-source alternative to Notion. You are in charge of your data and customizations. Built with Flutter and Rust.
GNU Affero General Public License v3.0
1.1k stars 228 forks source link

chore: store pending collab writes in memory #1000

Closed Horusiath closed 1 week ago

Horusiath commented 1 week ago

This PR replaced Redis pending collab writes queue with in-process channel. There are few reasons for that:

  1. If we want to scale out number of web service instances in the future they shouldn't override collab request: which is what the current implementation would do.
  2. It's much faster this way as there are no extra network hops.
  3. We don't need strong persistence of states: since server is not the source of truth, we don't need to worry what will happen when the collab is not persisted. If we ie. restart server, when a client connects, it will simply resend collab's missing updates.
appflowy commented 1 week ago

When a user wants to confirm whether the server has saved their data, do we need to query the database? Will it take too long for the user to notice that the data has been saved?

Horusiath commented 1 week ago

On the current main branch there's no such confirmation. In this branch, if we use save collab with write_immediate option it will try to write collab down immediately (without using a queue) and return when it's completed.

I was thinking about adding something like Option<tokio::sync::oneshot::Sender<Result<()>>> as a method param that would be confirmed when persister loop actually manages to save the collab.

appflowy commented 1 week ago

Before implementing a queue, we encountered a pool timeout issue. For instance, simultaneous writes or other API endpoints occupying pool connections could lead to pool timeouts. If pool timeouts occur more frequently, how can we address this issue?