mrc-ide / rrq

:runner::runner::runner: Lightweight Redis queues
https://mrc-ide.github.io/rrq/
Other
24 stars 4 forks source link

mrc-2683: allow retrying of tasks #89

Closed richfitz closed 1 year ago

richfitz commented 1 year ago

This PR is part 1 of proper fault tolerance: the ability to retry a task. The idea is explained hopefully reasonably in the vignette

The basic idea is:

With this in place there's a bunch of stuff we can implement later, including things like retrying up to n times after specified failure conditions. That can be saved into the task itself and the retry can be done by the worker at the point it saves the task result (or at the point where we deal with orphan tasks).

The situation with tasks that have dependencies is much more complicated and I've left it out of this PR.

Apologies that this is a bit big, but it's not really as bad as it looks from the count as we use precompiled vignettes - there are less than 400 lines of implementation here and much of that is passing extra args around!

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (c960ef7) 99.94% compared to head (d0f6f10) 99.94%.

:exclamation: Current head d0f6f10 differs from pull request most recent head 302d07f. Consider uploading reports for the commit 302d07f to get more accurate results

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #89 +/- ## ======================================== Coverage 99.94% 99.94% ======================================== Files 28 28 Lines 1797 1930 +133 ======================================== + Hits 1796 1929 +133 Misses 1 1 ``` | [Impacted Files](https://codecov.io/gh/mrc-ide/rrq/pull/89?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mrc-ide) | Coverage Δ | | |---|---|---| | [R/common.R](https://codecov.io/gh/mrc-ide/rrq/pull/89?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mrc-ide#diff-Ui9jb21tb24uUg==) | `100.00% <ø> (ø)` | | | [R/bulk.R](https://codecov.io/gh/mrc-ide/rrq/pull/89?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mrc-ide#diff-Ui9idWxrLlI=) | `100.00% <100.00%> (ø)` | | | [R/keys.R](https://codecov.io/gh/mrc-ide/rrq/pull/89?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mrc-ide#diff-Ui9rZXlzLlI=) | `100.00% <100.00%> (ø)` | | | [R/rrq\_controller.R](https://codecov.io/gh/mrc-ide/rrq/pull/89?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mrc-ide#diff-Ui9ycnFfY29udHJvbGxlci5S) | `99.83% <100.00%> (+0.04%)` | :arrow_up: | | [R/time.R](https://codecov.io/gh/mrc-ide/rrq/pull/89?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mrc-ide#diff-Ui90aW1lLlI=) | `100.00% <100.00%> (ø)` | | | [R/utils.R](https://codecov.io/gh/mrc-ide/rrq/pull/89?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mrc-ide#diff-Ui91dGlscy5S) | `100.00% <100.00%> (ø)` | | | [R/worker\_run.R](https://codecov.io/gh/mrc-ide/rrq/pull/89?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mrc-ide#diff-Ui93b3JrZXJfcnVuLlI=) | `100.00% <100.00%> (ø)` | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mrc-ide). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mrc-ide)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.