Closed bigjools closed 4 months ago
Maybe try this.
Modify the Worker class in spinach to handle non-idempotent jobs that fail on a dead broker. Specifically, you can add a check in the process_job method to see if the job has max_retries unset, and if so, signal the job as failed if it fails due to a dead broker.
Here's an example implementation:
from spinach.worker import Worker
class CustomWorker(Worker):
def process_job(self, job):
try:
self._process_job(job)
except Exception as e:
if not job.max_retries and 'dead' in str(e):
self.send_failure_signal(job)
else:
raise e
Fixed in 0.0.23
Work was recently completed to send signals for jobs that fail as a result of a dead broker, however those jobs that have max_retries unset are not signalled as failed at all, unlike the idempotent ones which DO send a failure signal.