Implement an initial background worker / job queue system

[lmorchard]

This thing needs a system to run deferred tasks in the background. It can be very simple at first - e.g. just an event responder that can run async in the same process as web server? Maybe better to start as a second process?

Let's see how hard it ends up being to deploy. Also see how little of the wheel we need to re-invent to keep it initially simple.

[lmorchard]

Would also be great if this thing had some facility for reporting progress back to the initiator of the task - e.g. for import UI (#17) it would be nice to kick off async background import while reporting periodic percentage completion to user

Maybe construct a special log stream listener that can route events back to a user-accessible channel / websocket that narrates to user?

[lmorchard]

Maybe use Bree for background workers with their own job management rather than something more of a generic job queue?

Check out JobManager in Ghost, which uses Bree. Also:

• job-service.js
• import-manager.ts
• update-check.js

[lmorchard]

Looks like Bree is more focused on task scheduling? I might want something more like a worker thread pool, like Piscina:

• https://piscinajs.github.io/piscina/

Thinking about how the communication would work with SQLite being a single-writer scenario as a repository. Maybe worker threads can send write requests back up to the parent process to be enqueued for handling

[lmorchard]

Thinking through a few things here:

Considering the "simplest" deployment scenario with SQLite and zero dependencies in external services - i.e. no postgres, no redis, no queue.

The primary web head VM also has to serve as the sole worker VM. Only it can make writes to SQLite, and then only in one process - no parallel processes for writes. Could handle this by running everything inside the same process. But also could handle this by using worker processes that post messages back to the primary process to perform DB writes.

Can trade a bunch of these worries for external dependencies - i.e. instead of single-writer SQLite, switch to postgres and instead of in-process queue processor switch to something using redis. Consider that a more complex deployment scenario that can scale better. Thinking we need a bunch of users first before worrying about that, though