SQLite worker queue implementation

[isaacdonaldson]

This is an implementation of an SQLite background worker queue.

Since SQLite does not have FOR UPDATE SKIP LOCKED and sqlx does not support BEGIN IMMEDIATE transactions (from what I could tell), I opted to use another table aquire_queue_write_lock as concurrency control.

When reading for dequeue, it will attempt to update a field in the table, on success it will turn the transaction into a write transaction (from the starting point of read transaction). This will prevent the other readers from reading a queued task until the current one is done selecting one. On failure it will not select a task as there is another transaction selecting one, making sure that no task is selected twice.

[jondot]

wow i was hoping that someone with sqlite expertise come in and implement this one! thank you!! i'm not an expert but I've just spent some time reading about other sqlite implementations, and it looks like people say that sqlite locks the entire DB on writes. would that obsolete the locking table/issue?

[isaacdonaldson]

It does lock on writes, but the dequeue method is a read transaction until it finds a task, then afterwords when claiming it, it gets upgraded to a write transaction. The write into the lock_queue will upgrade the transaction to a write queue before it fetches a task, so it will be locked for the read and remove the chance a task is claimed twice.

[jondot]

Makes sense. Fantastic contribution ❤️