closeio / tasktiger

Python task queue using Redis
MIT License
1.42k stars 80 forks source link

Add functionality to purge periodic task execution histories #151

Open jkemp101 opened 5 years ago

jkemp101 commented 5 years ago

https://github.com/closeio/tasktiger/issues/137 addresses purging tasks in an error state. But long running periodic tasks can have an ever increasing list of execution errors that will never get purged. A function should be provided that can be used to trim the execution list for periodic tasks so that they don't have unbounded growth.

Another idea would be to have a max executions config setting and do a LTRIM every time we add to the execution list here: https://github.com/closeio/tasktiger/blob/fe82e842f42b28c46c265f4b54de0f776f776404/tasktiger/worker.py#L409-L412

AlecRosenbaum commented 4 years ago

Another use case that could create execution history issues is for a task that has many retries before failing.

For example, say there's a queue that delivers many small payloads to https://app.example.com. If that website become unavailable and the delivery task retries every minute for a day, it's possible to end up with over a thousand recorded executions for just one task. Each execution record takes up roughly as much space as the queued task, so 1 bad task could end up consuming over a thousand tasks worth of space.

For just one task it's not that bad but if thousands of tasks all fail the same way, it's easy to run out of memory in redis.

I think the best solution here is adding a max executions setting, then doing an LTRIM if it's set.

thomasst commented 3 years ago

We should then store the number of executions in a separate Redis key (or in the JSON payload of the task).

Another issue is that unique or periodic tasks share one execution history. For example, if you have a periodic task that retries up to N times, then the N retries don't currently apply to each period -- they apply to the overall lifetime of the periodic task, which isn't what we want. That's because we store each failed execution under the task key and don't clean them up.