Note that there is also a "missed heartbeat from celery" log message that may or may not be related.
Important:
[ ] I added a 99% exclusion rule related to the "missed heartbeat from celery" logs, and provided details in https://2u-internal.atlassian.net/browse/GSRE-1952?focusedCommentId=5146559. If GSRE-1952 fixes the drift issue, and the missed heartbeat turns out to be a separate problem, the missed heartbeat issue should be ticketed separately and not lost with this ticket. The retention rule could be switched to point to the updated ticket as well.
We are seeing large numbers of celery errors with "Substantial drift from celery" warnings:
Important: