Our homegrown backoff/retry was good enough for a while, but we ran into a huge disabling event when too many changes were made simultaneously. Since this code was first written, celery has built in good back off/retry functionality, including jitter, to make sure that all the retries don't happen simultaneously.
Switches to using celery native backoff
Refactors a huge try/catch block so the exception handling is on smaller subsets of code
FIXES: APER-3510
Testing instructions
Using a course with many enrollees, change a course overview in a way that will trigger a new run of award/revoke course certificates.
Description
Our homegrown backoff/retry was good enough for a while, but we ran into a huge disabling event when too many changes were made simultaneously. Since this code was first written, celery has built in good back off/retry functionality, including jitter, to make sure that all the retries don't happen simultaneously.
FIXES: APER-3510
Testing instructions
Using a course with many enrollees, change a course overview in a way that will trigger a new run of award/revoke course certificates.