Closed ccostino closed 4 weeks ago
This happened many times on one day back on June 18th, and again (similar to other exceptions reports) when you drill down to the details of what was going on at the time on the system, the first thing you see in the chain of events is:
[2024-06-18 21:11:10 +0000] [7] [ERROR] Worker (pid:23741) was sent SIGKILL! Perhaps out of memory? [2024-06-18 21:11:10 +0000] [23752] [INFO] Booting worker with pid: 23752
I think the sequence of events might have been:
I recommend we set memory to at least 4 gb on production.
@stvnrlly and I talked about this today, we're planning on doubling the memory in production, which would put it at 4 GB.
We ended up not increasing the memory, however we did increase the number of worker instances for the celery workers in production. This should help greatly and especially in cases where the app is being redeployed or the platform is undergoing maintenance and the apps are restaged under the hood.
If we see this pop up again and additional errors like Worker (pid:23741) was sent SIGKILL! Perhaps out of memory?
, then we'll need to do another memory increase with what we can.
This is one of the errors we've seen captured in New Relic that we'd like to dig into and understand, if not also resolve.
Error message: RETRY FAILED: Max retries reached. The task send_email_to_provider failed for notification XXXXXXXXXXXX. Notification has been updated to technical-failure Exception: app.exceptions:NotificationTechnicalFailureException
Implementation Sketch and Acceptance Criteria
Security Considerations