samuelcolvin / arq

Fast job queuing and RPC in python with asyncio and redis.
https://arq-docs.helpmanual.io/
MIT License
1.98k stars 165 forks source link

Use function's keep_results configuration when storing failed jobs results #433

Open joshwilson-dbx opened 3 months ago

joshwilson-dbx commented 3 months ago

Taking a stab at using a function's (or cron job's) configuration for keeping the job results when the job fails.

I don't think this is a complete solution yet. There are a few cases where the job results are written to redis using the Worker's configuration because we can't get the Function in order to use it's config.

But maybe this is a situation where it's good enough to improve the situation for the most common situation?

Addresses #417 Relates to #416

codecov[bot] commented 3 months ago

Codecov Report

Merging #433 (a06ea04) into main (ab2dda2) will decrease coverage by 0.37%. The diff coverage is 73.33%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #433 +/- ## ========================================== - Coverage 98.40% 98.03% -0.37% ========================================== Files 11 11 Lines 1063 1071 +8 Branches 200 205 +5 ========================================== + Hits 1046 1050 +4 - Misses 8 10 +2 - Partials 9 11 +2 ``` | [Files](https://app.codecov.io/gh/samuelcolvin/arq/pull/433?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Samuel+Colvin) | Coverage Δ | | |---|---|---| | [arq/worker.py](https://app.codecov.io/gh/samuelcolvin/arq/pull/433?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Samuel+Colvin#diff-YXJxL3dvcmtlci5weQ==) | `97.80% <73.33%> (-0.78%)` | :arrow_down: | ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/samuelcolvin/arq/pull/433?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Samuel+Colvin). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Samuel+Colvin) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/samuelcolvin/arq/pull/433?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Samuel+Colvin). Last update [ab2dda2...a06ea04](https://app.codecov.io/gh/samuelcolvin/arq/pull/433?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Samuel+Colvin). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Samuel+Colvin).
JonasKs commented 2 months ago

Agree, I think this looks good. 😊

epicwhale commented 1 month ago

Thanks for looking at this one! Any chance this can enter the v0.26.0b1 release? :)