Open dmitry opened 7 years ago
passenger_memory_limit
is the maximum amount of memory that an application process may use, in megabytes. Once an application process has surpassed its memory limit, Passenger will allow it to finish processing all of its current requests, then shut the process down. However, shutting down happens according to how your application is written, and can take some time, this is why you see the increased number of processes. If you profile your application while it is shutting down you may be able to speed this process up, however it is more important to avoid having to replace processes all the time, so you will want to evaluate whether your memory limit is reasonable.
Because you have many threads, you may want to consider the expected memory footprint of your application. It may be that 700MB is too low. If your single-threaded memory footprint is 200MB we estimate that each thread will use an additional ~10% of the total memory footprint or ~20MB of RAM, that means that for 30 threads you would expect to see 200_(1+(0.1_30)) = 800MB of ram quite easily.
There is some helpful information on choosing your configuration values in this article: https://www.phusionpassenger.com/library/config/nginx/optimization/.
If you find that 700MB for 30 threads should suffice for your application, then profiling your application for leaks would be the next step.
Yes, I tried to profile for ruby memory leaks - but we are free of them; seems like all the leaks somewhere inside the gems, in C extensions... at the moment I don't have much time to analyze and solve those issues, so the only solutution at the moment to have passenger_memory_limit
enabled.
The problem I can see at the moment, that Passenger tries to spawn new workers too early, before actually queue of other workers will be disconnected from the traffic and shutdowned. Means when you have room of 32GB on hardware machine, and you have 24 instances with 1GB each, passenger may kill your server with swapping because it can create additional 24 instances, so in total all of them could take up to 48, or even more, when memory limitation of the old processes are larger than 1GB (that's why actually they are going to be killed).
Additionally, I noticed that passenger may leave unlimited number of processes on disabled state, so in total number of processes may be upper than 48 in total.
Any chance to get an option to limit the passenger memory?
While working in
thread
concurrency model number of workers are growing, and unable to be disabled even after memory limitation should get out of the traffic stream. Example below is not perfect, as I've copied it in the third wave when tried to choke the fire.Also that means there could be dozen of disabled workers, which are still in working state. Means we should prepare for
8 (passenger_max_instances) * 800 (passenger_memory_limit + 100mb) * 2 (disabled and enabled)
memory or even more, as number of disabled could be grown to unlimited number of workers.Please note
passenger_max_request_time
is zero, as limitation inthread
concurrency model kills process with all the concurrent processes (see https://github.com/phusion/passenger/issues/1850).Passenger config, most important settings:
passenger-status
shows workers are growing, I saw even more ~15 of additional workers (in the first wave), that tried to be disabled.PS. Is there are a suggestions how to setup thread applications?
passenger enterprise 5.0.30/nginx ubuntu 14.04 APT repo Ruby 2.1.7, rbenv, Rails 3.2.2.22 Plain hardware used