search

cenkalti / kuyruk

⚙️ Simple task queue for Python
https://kuyruk.readthedocs.org/
MIT License
234 stars 17 forks source link

Add a delay between retries when task crashes #49

Closed frol closed 7 years ago

frol commented 7 years ago

Currently, it seems that there is no delay at all between retries when my task raises an exception. Sometimes bugs occur in the tasks logic and they simply raise an exception. I don't want to lose any of the tasks, so I went trying to set retries=2**16, but it retries right away.

What would be the recommended way of handling this use-case?

Catching all exceptions inside the tasks and sleep doesn't make a lot of sense to me as it will be wasting time sleeping instead of processing other tasks...

cenkalti commented 7 years ago

@frol @khorolets I have build a feature for delaying failed and rejected tasks: https://github.com/cenkalti/kuyruk/commit/d63b1483f3bd93c96c9a38e5527feef60cbcd596

If you set fail_delay parameter on the task re-queuing of the message is delayed for specified seconds.

My suggestion is to set fail_delay and leave retrying operation to the broker. Does that solve the issue?

khorolets commented 7 years ago

@cenkalti Looks awesome for me! @frol what do you say?

frol commented 7 years ago

@cenkalti This is exactly what I was looking for! Thank you for this blazing fast response with implementation!

cenkalti commented 7 years ago

New version is available on PyPI: https://pypi.python.org/pypi/Kuyruk/8.2.0

khorolets commented 7 years ago

Thank you!

frol commented 7 years ago

@cenkalti It would be nice and consistent if you also create a tag in the repo for 8.2.0 release.

cenkalti commented 7 years ago

@frol I have pushed the tags, sorry about the delay.

frol commented 7 years ago

@cenkalti No problem at all. That was just a friendly reminder.