Delayed Jobs and Retry Jobs with Backoff

[thoven87]

I wanted to open this ticket to discuss the proposal for adding both delayed and retry jobs with exponential backoff.

I think both delayed and retry with backoff are similar in that they can be scheduled to be executed at a later date.

Looking at the diagrams below, I think we can put both features in the same category.

The Postgres implementation is the easier out of three JobQueue drivers currently available (Memory, Postgres, Redis)

1 - Postgres Driver:

I am proposing of adding one more column to the _hb_pg_jobs table called scheduled_at with Nullable value.

2 - Redis Driver:

This will require one of two new row keys swift_jobs::scheduled::${JobID} and swift_jobs::ready${JobID} we can use Redis scripts to atomically move jobs from state (scheduled) to another state(ready). I am not too familiar with the RedisStack yet to know of this limitation.

I welcome some feedback on how to proceed further @adam-fowler @Joannis

[adam-fowler]

I think you can build the retry with back off on top of delayed jobs. That is why I was suggesting we do a PR for delayed jobs first. Once we are happy with that we can implement the retry with back off.

Regarding the implementation we would need to separate the delay out from the data passed to the job driver as the different drivers will need access to this value.

[adam-fowler]

Regarding drivers, yeah Postgres is easy. I also think the memory one is easy enough as well. A Redis script could be used for moving delayed jobs, but we'd have to be careful it doesn't lock the database too long. I am also a little concerned about moving logic onto the redis server.

[thoven87]

First pass of delayed job implementation here I will follow up with the Postgres Driver then Redis.

[thoven87]

I think we can close this ticket now!