The current implementation of the queues has some issues:
Prone to deadlocks
Doesn't handle request spikes
Code duplication between CacheAdapters
Doesn't support multiple apps using the same cache adapter.
Horizontal scaling
Parallelisation and cpu-heavy work offload to background threads/processes will allow the Document Drive to scale.
The main thread shouldn’t block because 10k operations are being serialized and hashed.
Queues seem like the best place to add this. Due to the nature of the sync mechanism, even if there are thousands of operations waiting to be added to a document, operations to a different document can be performed at the same time.
Synchronization units are a prime target for work distribution but they have dependencies between each other. CREATE_DOCUMENT needs to wait for ADD_FILE on Document drive queue.
Multiple instances of the host app (Switchboard) should only be needed when code external to the Document drive, like Graphql request handling, become the bottleneck.
Optimistic concurrency control
Prisma has a good write-up on this: https://www.prisma.io/docs/orm/prisma-client/queries/transactions#read-modify-write
Currently, we have a pessimistic control where a worker waits for a lock on the db and performs all the work inside a db transaction to make sure it is accessing the most up-to-date data. This creates long locks and adds load to the db.
Taking an optimistic approach involves a worker reading the latest state (without locking), applying the new operations and trying to write them to the db.
If other operations were added in the meantime then it will fail. If it fails then the worker will just redo the process on the new state.
This makes it so that the CPU heavy tasks like serializing and hashing the state, reshuffling operations, running the reducers, etc, are performed on background workers.
We control what operations are applied using the queue manager so conflicts should be rare.
Technical issues
If we use different processes then all communication between the queue manager and the workers has to be through serialized objects. Currently, we pass a callback to a worker so it can write to the db, this is no longer possible.
One simpler alternative is for the worker to return the operations it wants to store and the QueueManager performs the call to the DB.
This is not as scalable since everything has to go through the QueueManager (running on the main thread).
Each worker could have it's own connection to the cache and storage layers, allowing it to bypass the QueueManager to complete a job.
For example, the current Postgres instance we are using in productions supports 20 connections. This would move the parallel load to the services we use (Redis/Postgres), outside of Switchboard.
We could even have these workers running in the cloud with serverless computing.
Node supports multiple processes with Cluster and multiple threads with Child process.
In the browser, parallelisation can be added by using Service Workers.
Tasks
Implement base CacheManager to promote code reuse. Cleanly separate what's kept in memory and what's kept in the cache adapter.
Use events instead of infinite loop. A Worker should be able to request a new job when it's done and listen for new jobs.
Built-in support for separate processes. Cluster in Node and Service workers on Browser.
Create a small subset of document drive with direct access to the db and cache to be used by each worker.
Use optimist updates. Rely on DB for rejecting conflicting operations. If it fails then recalculate operations and retry.
Handle conflict errors when performing operation. (Exponential back offs, when to give up? The worker could try to retry until a certain limit and return job to manager for it to go back to the queue of jobs to be picked up. It would be moved to a wait_to_retry state with a timestamp for the next retry.
Manager could keep track of the timestamp of the next job to retry and set a timeout to add it back to the queue.
Research was done on the API's
One queue vs Multiple queues
The current implementation of the queues has some issues:
Horizontal scaling
Parallelisation and cpu-heavy work offload to background threads/processes will allow the Document Drive to scale. The main thread shouldn’t block because 10k operations are being serialized and hashed. Queues seem like the best place to add this. Due to the nature of the sync mechanism, even if there are thousands of operations waiting to be added to a document, operations to a different document can be performed at the same time. Synchronization units are a prime target for work distribution but they have dependencies between each other.
CREATE_DOCUMENTneeds to wait forADD_FILEon Document drive queue.Multiple instances of the host app (Switchboard) should only be needed when code external to the Document drive, like Graphql request handling, become the bottleneck.
Optimistic concurrency control
Prisma has a good write-up on this: https://www.prisma.io/docs/orm/prisma-client/queries/transactions#read-modify-write Currently, we have a pessimistic control where a worker waits for a lock on the db and performs all the work inside a db transaction to make sure it is accessing the most up-to-date data. This creates long locks and adds load to the db.
Taking an optimistic approach involves a worker reading the latest state (without locking), applying the new operations and trying to write them to the db. If other operations were added in the meantime then it will fail. If it fails then the worker will just redo the process on the new state. This makes it so that the CPU heavy tasks like serializing and hashing the state, reshuffling operations, running the reducers, etc, are performed on background workers.
We control what operations are applied using the queue manager so conflicts should be rare.
Technical issues
If we use different processes then all communication between the queue manager and the workers has to be through serialized objects. Currently, we pass a callback to a worker so it can write to the db, this is no longer possible.
One simpler alternative is for the worker to return the operations it wants to store and the QueueManager performs the call to the DB. This is not as scalable since everything has to go through the QueueManager (running on the main thread).
Each worker could have it's own connection to the cache and storage layers, allowing it to bypass the QueueManager to complete a job. For example, the current Postgres instance we are using in productions supports 20 connections. This would move the parallel load to the services we use (Redis/Postgres), outside of Switchboard. We could even have these workers running in the cloud with serverless computing.
Node supports multiple processes with Cluster and multiple threads with Child process. In the browser, parallelisation can be added by using Service Workers.
Tasks
wait_to_retrystate with a timestamp for the next retry.Research was done on the API's One queue vs Multiple queues