search

phusion / passenger

A fast and robust web server and application server for Ruby, Python and Node.js
https://www.phusionpassenger.com/
MIT License
5.01k stars 548 forks source link

Out of memory leads to passenger shutdown and ensuing 502 error on nginx #2526

Closed mamsyscheezoo closed 9 months ago

mamsyscheezoo commented 9 months ago

This is the code snippet from the Nginx logs.

out of memory

This indicates a memory issue, which caused passenger to shut down, eventually leading to server downtime.

We use certificates issued by Let's Encrypt, and these certificates are automatically renewed using a software called Certbot. Once the certificates are auto-renewed, Certbot usually reloads nginx automatically. This is because nginx does not read the certificate off disk every time for performance reasons. Our certificates are valid for 3 months, but we renew them every 2 months. Normally, nginx is reloaded without causing downtime. However, in the latest renewal attempt, Certbot was unable to reload the nginx service, resulting in server downtime. We suspect that this was due to "Out of Memory Issue".

How can the memory issue be tracked down? And is it certain that the nginx failed to reload due to Memory issues? Also if the perl module is out of memory? How can we debug it and track the root cause.

Any detailed help would be great. Please.

CamJN commented 9 months ago

Since the issue is not with passenger, this is the wrong place to ask.