openedx / edx-platform

The Open edX LMS & Studio, powering education sites around the world!
https://openedx.org
GNU Affero General Public License v3.0
7.09k stars 3.79k forks source link

feat: switching to celery native backoff for cert awarding #35009

Closed deborahgu closed 6 days ago

deborahgu commented 2 weeks ago

Description

Our homegrown backoff/retry was good enough for a while, but we ran into a huge disabling event when too many changes were made simultaneously. Since this code was first written, celery has built in good back off/retry functionality, including jitter, to make sure that all the retries don't happen simultaneously.

FIXES: APER-3510

Testing instructions

Using a course with many enrollees, change a course overview in a way that will trigger a new run of award/revoke course certificates.

edx-pipeline-bot commented 6 days ago

2U Release Notice: This PR has been deployed to the edX staging environment in preparation for a release to production.

edx-pipeline-bot commented 6 days ago

2U Release Notice: This PR has been deployed to the edX production environment.