timgit / pg-boss

Queueing jobs in Postgres from Node.js like a boss
MIT License
2.15k stars 160 forks source link

Add a way to limit maximum retry delay for exponential backoff #441

Open KristjanTammekivi opened 7 months ago

KristjanTammekivi commented 7 months ago

Hi, I would like for there to be a way to adjust maximum retry delay or configure it in some other way since even with minimum retry delay the exponential backoff sets a very large delay.

For instance with 1 second delay

postgres=# SELECT
    (
        1 * 2 ^ LEAST(16, i + 1) / 2 + 1 * 2 ^ LEAST(16, i + 1) / 2 * random()
    ) * interval '1'
FROM generate_series(1, 16) AS t(i);
    ?column?     
-----------------
 00:00:02.166215
 00:00:07.481996
 00:00:10.5314
 00:00:18.985739
 00:01:01.490028
 00:01:25.182171
 00:03:08.251286
 00:06:50.541206
 00:16:33.832786
 00:23:04.647317
 00:35:50.171085
 01:37:15.770498
 02:35:16.779842
 07:22:58.370967
 10:40:32.358324
 14:41:46.530635
(16 rows)

When using a 5 second retryDelay this gets even wilder with the last delay being constantly in the range of 40-90 hours

I'd like there to be some more tunability for this, maybe so we can configure the maximum power (currently 16), allow setting a maximum retryDelay, or allowing to pass in a custom SQL to decide that.

I know this was quickly discussed in https://github.com/timgit/pg-boss/issues/340 but I do think this should be reconsidered

timgit commented 1 month ago

This should be addressed in v11