search

quirrel-dev / quirrel

The Task Queueing Solution for Serverless.
https://quirrel.dev
MIT License
892 stars 67 forks source link

Question on job update #63

Closed sehmaschine closed 3 years ago

sehmaschine commented 3 years ago

I've recently asked this on Twitter (see https://twitter.com/sehmaschine/status/1350415050822066176).

We have different kinds of background tasks with our appliations:

  1. Background tasks which are independent of a user action, e.g. sending daily mails or doing backups. These tasks are usually part of a distinct application/microservice. I can see how these tasks are handled with Quirrel.
  2. Long running jobs which have to run in the background, e.g. generating PDFs. These tasks are part of the main app and are triggered by a user action (e.g. clicking a button). Then the user either stays on the page and needs updates on the process or the user leaves the page and we will inform her about the process later on. With this usecase, we need to update a single job with the current state (e.g. 23 of 80 PDFs created) regularly.

When looking at the Quirrel pricing page (which refers to API calls), it seems that usecase 2 has not been a major concern, right? Because in that case I would (for example) have 100 API calls for 1 process (because of the in-between process updates). I just wanna make sure we are not backing the wrong horse ...

Skn0tt commented 3 years ago

As originally answered here

Hi @sehmaschine! Quirrel focuses on the scheduling (as in "wake up your application when a job is due") and distribution (as in "spread out work across multiple lambdas to speed up") part of job queueing, and is very similar to Repeater in that regard. It does not aim to keep track of state like "number of PDFs processed", and there are no immediate plans to change that.

I'd recommend handling the PDF-Progress state outside of Quirrel. If it's ephemeral, you could just publish updates using an approach similar to the one demonstrated here: https://demo.quirrel.dev/distribute

On the pricing model: I decided to go with API-Call-Based pricing since it's closely-tied to my own operating cost. I totally get that this doesn't work for some applications - feel free to reach out to info@quirrel.dev and I'm sure we'll figure out something that works for both of us. And: Quirrel is MIT-Licensed, so you're totally free to host your own instance!