celery / django-celery-results

Celery result back end with django
Other
684 stars 203 forks source link

Add retry logic for Django and Psycopg2 InterfaceErrors #290

Closed juspence closed 2 years ago

juspence commented 2 years ago

This PR implements https://github.com/celery/django-celery-results/issues/119 in a slightly different way. It adds retry logic only for django.db.utils.InterfaceError and psycopg2.InterfaceError, which can be errors like "connection already closed".

It does not retry any DatabaseError, which can be errors like "numeric value out of range" or "foreign key check failed". See https://www.psycopg.org/docs/module.html#exceptions for more info.

exception_safe_to_retry() is overridden from the parent class: https://docs.celeryproject.org/en/stable/_modules/celery/backends/base.html#BaseBackend

This method is called when an exception is raised in _store_result() and always_retry / result_backend_always_retry is set to True. If the exception is an InterfaceError, the connection is closed as a side effect of calling exception_safe_to_retry().

Then the retry logic implemented in the parent class continues, and Django will automatically open a new connection for the next attempt to store the task result or get the task metadata.

auvipy commented 2 years ago

I want to add this to next release, can you please address the review comment?

juspence commented 2 years ago

@auvipy Yes, I am working on this today since I had other tasks to finish Friday.

auvipy commented 2 years ago

ok