Investigate and fix the staging site

[ccostino]

Our staging site is currently returning a 502: Bad Gateway error and it's unclear why given that all recent merges went in clean, the site deployed to staging just fine, and things run fine locally.

The only clue at the moment is that the logs for the admin site keep showing this error (shortened for brevity):

[13] [ERROR] Worker (pid:28830) was sent SIGKILL! Perhaps out of memory?
[28840] [INFO] Booting worker with pid: 28840
[13] [ERROR] Worker (pid:28831) was sent SIGKILL! Perhaps out of memory?
[28841] [INFO] Booting worker with pid: 28841
[13] [ERROR] Worker (pid:28832) was sent SIGKILL! Perhaps out of memory?
[28842] [INFO] Booting worker with pid: 28842

The API appears to be completely fine so it seems this is an issue with just the admin, but that's not 100% clear yet.

Implementation Sketch and Acceptance Criteria

• [x] Dig into the logs a bit more for the admin site and see what might be causing these processes to continuously be terminated
  • Either https://logs.fr.cloud.gov/ or in the command line, e.g., cf logs notify-admin-staging
• [x] Figure out what might be the cause for the errors and attempt to trigger them locally
• [x] Check to make sure that there is enough memory available for the app in cloud.gov
  • The app is deploying fine, so presumably yes because there's overhead free for the deployment to happen
• [x] Double check that the staging API is functioning properly as well

Security Considerations

• Be mindful of what logs are pasted either here or in Slack when asking for help and sharing information; make sure any sensitive info is removed first!
• We cannot deploy to our production environment until we understand and fix what is happening in our staging environment.

[jskinne3]

Staging restored! I increased the memory quota of notify-admin-staging (here) from 1024 to 2048 (one gig to two gigs). The app restarted and works. Check https://notify-staging.app.cloud.gov/

While it was down:

• I found the Log Stream feature of Cloud.gov was very useful! ✨
• I observed that the Staging logs were showing furious activity. Workers constantly booted and killed, without any incoming web requests. By contrast, Demo logs were calm.
• Staging boots 17 workers (PIDs 112–128) whereas Demo boots only 5 workers (PIDs 112–116) upon startup.
• I captured these log snippets:
  • notify-admin-demo.log orderly startup
  • notify-admin-staging.log furious startup

[terrazoon]

We are going to adjust the number of workers in the gunicorn config. See the linked issue.