Add a queue management system to perform long-running async jobs

[FelixMalfait]

We are performing various task across the app such as sending notifications which would be better handled asynchronously.

We need to add a system to manage jobs/queues.

NestJS has a built-in package that is a wrapper around BullMQ/Bull. We should use BullMQ and not Bull (Bull is in maintenance mode)

One additional requirement is that we don't want to make setting up this app locally more complex, so if QUEUE_DRIVE=sync (default value in .env.example) then the job should be processed immediatly as it is created.

[FelixMalfait]

I like this idea of a Postgres-based queue: https://github.com/timgit/pg-boss

[brendanlaschke]

I think with https://github.com/apricote/nest-pg-boss it should be really easy to implement. We would only need to create the db/schema to store the queues and add the setup for each job type (notifications, workspace deletion, ... ).

But I think this is would be best handled with the first ticket in need of the queueing system?

[FelixMalfait]

@brendanlaschke I didn't look into it into details but this PR from @martmull could be a good candidate to refactor with queues > https://github.com/twentyhq/twenty/pull/2132 (webhooks/triggers should be performed asynchronously)

[brendanlaschke]

Oh yes interesting one, now I don't think it is that good of a candidate as we are handling an external request - I experimented with a queue and am kind a stuck on one problem:

1. There is an incoming request for e.g. the delete hook

   deleteOneHook(@Args() args: DeleteOneHookArgs): Promise<Hook>

2. This is then added to a queue
3. Later handle the job from the queue

Currently at 2. the hook promise would be returned (prismaService.client.hook.delete(...)) - wich would as I understand it also return some failed status if the deletion fails.

But now adding the job to the queue and handling this queued job at some point later we cannot respond with a correct status? Instead there could be only a "job has been queued" response?

As you as a team have a bit more experience as I have - have I missed something here or do you have an idea to solve this by using a different structure?

[FelixMalfait]

@brendanlaschke I'm sorry I think I led you into the wrong direction. I read the PR titled "first trigger" then quickly saw "Triggers when a new company is created" so I assumed that we had implemented our very first hook entirely. But looking at the code it seems that we have only implemented the first part to register a hook, but not the mechanism that dispatches an event to the registered webhook urls. (cc @martmull correct me if I'm wrong).

The typical use-case for a queue system is when you have to make an http request to an external service. Because you don't have control over the response time of that external service, you don't want to make your own response time dependent of it, so in that case "job has been queued" is indeed considered good enough to return a 200 immediately. Then it's the role of the queue management system to monitor the job execution and let someone knows if things didn't go as expected.