Provide a config option to never fail a job with an `Job exhausted retries in queue`

[myronmarston]

We've opted to use a ruby qless middleware to handle retries rather than doing it in the qless-core scripts:

https://github.com/seomoz/qless/blob/master/lib/qless/middleware/retry_exceptions.rb

This is preferrable for us because "Job exhausted retries in queue" gives nowbacktrace or indication about what the original failure was -- so it hides the source of the problem from us. By doing it in ruby, we get the full backtrace and can troubleshoot the problem. In addition, if we call Job#retry, we want it to retry: we don't qless-core failing it instead with this error.

Currently we're running qless-core and qless branches that have this feature disabled but we don't want to stay on a branch as it makes getting updates hard:

https://github.com/seomoz/qless/tree/kill_exhausted_retries_error https://github.com/seomoz/qless-core/tree/myron-disable-exhausted-retries-error

[dlecocq]

• Allow retry to provide a group, message like fail
• Provide a pubsub event and configurable grace period for timeouts

[dlecocq]

Just as a note to myself, it seems like to really get this done correctly, I should provide the grace period for each timeout.

[dlecocq]

Phew! That was a tricky one, but it's implemented

[dlecocq]

It seems to me that with the advent of group and message in the retry call, as well as support for a grace-period, we've eliminated the need for not exhausting retries for a job? Go ahead and reopen if I've misunderstood, though.

[myronmarston]

Yep, things are working fine for us now.