RFC: Queue System

[zomars]

Is your proposal related to a problem?

Right know we depend on many external factors that may affect our core functionality in terms of speed, performance and reliability. Some examples that come to mind are:

• Sending emails
• Sending web hooks
• Sending broken integration alarms #2944
• Future SMS/notification sending #2861

Putting the example for the Webhooks, we don't have control on the reliability of those endpoint. Right now a failed web hook doesn't break the main core flow but a slow endpoint may affect the overall performance of the booking event.

Describe the solution you'd like

I'm thinking about Queue System that allow us to delegate certain tasks or actions so the main flow doesn't concern if the actions actually succeeds or not. We can use a Queue for many things like:

• Delegating non-critical tasks to the queue
• Delegation notifications
• Sending alerts to soon to expire trials
• Delegate sending emails
• Automatic retry for failed tasks
• Automatic alerts if a task failed X amount of times

Describe alternatives you've considered

I'm not aware of any SaSS alternatives but for our use case it should be fairly simple schema:

enum QueueTask {
  email
  notification
  webhook
}

model Queue {
  id            String    @id @unique @default(uuid())
  task          QueueTask
  payload       Json?
  lastAttemptAt DateTime?
  nextAttemptAt DateTime?
  attemptCount  Int       @default(0)
  createdAt     DateTime  @default(now())
  updatedAt     DateTime  @default(now()) @updatedAt
}

Additional context

• 2944

• 2861

[agustif]

Not that I'm against the proposed route of using what we already have for now adding those to prisma.schema

But wondered if we want to even consider adding Redis to the mix? which with something like BullMQ https://github.com/taskforcesh/bullmq to make it work nicely with node.... I guess a huge con of this approach would be the new stack requirement of redis for self-hosters and such. Not great.

As for SaaS, I knew of https://quirrel.dev/ which was integrated with vercel/nextjs before, but has been acquired by netlify it seems now?

https://docs.quirrel.dev/netlify-acquisition-faq tldr: sunsetting on july the service, not accepting new customers BUT

Not sure if it's FOSS we can deploy? https://docs.quirrel.dev/deploying/#deploy-the-image https://github.com/quirrel-dev/quirrel/pkgs/container/quirrel

In which case we might be able to offer a hosted solution for queue.cal.com (redis underneath) for self-hosters too, if they don't want to spin it themselves

All in all, we're probably good just using postgres via prisma as @zomars proposed in RFC, and just making sure to delete / prune the queues once data is stall / no longer relevant or such.

[zomars]

I'd heard good things about Bull. But for our initial use case could be over kill. Also as you mentioned if we don't want to complicate the stack for self hosted, we shouldn't depend on external SaaS either. @agustif