Patch: Limit Memory of an ApplicationServer Instance

[FooBarWidget]

From schmalowsky on February 12, 2009 11:18:40

I'd like to supply a patch to limit the max memory usage of an ApplicationServer instance.

Example: If you have set RailsMaxInstancesPerApp to 4 then each instance's memory is limited to PassengerMemLimit.

PassengerMemLimit can be given in (K|M)Bytes, e.g.

PassengerMemLimit 134217728 or PassengerMemLimit 131072K or PassengerMemLimit 128M

It would be great if this patch finds its way into the official source. And I hope that other might find it useful, too.

Attachment: PASSENGER-APPLICATION_SERVER-MEMORY_LIMIT.patch

Original issue: http://code.google.com/p/phusion-passenger/issues/detail?id=201

[FooBarWidget]

From nmeans on April 10, 2009 10:50:46

This should probably be tagged as "Enhancement" rather than "Defect", since it not being there doesn't really cause Passenger to fail in some way. Would definitely be a useful enhancement.

In the meantime, the "PassengerMaxRequests" setting comes in handy to address this problem. Just watch your Passenger processes and see how many requests it takes them to start getting too big, then set MaxRequests lower than that number and Passenger will kill the process off after it has served that number of requests. It's not as ideal as a mem limit would be, but it works for now.

[FooBarWidget]

From jamesrtkay on April 20, 2009 14:38:27

A strong "+1" from me. I recently had a defect which caused the full 16GB of my server to be occupied by just one of my Passenger Rails processes, forcing me to power-cycle it. I attempted to use "ulimit" which did not work. The patch here appears to be a workable solution and I think would be used by many. Thanks.

[FooBarWidget]

From woody.peterson on April 28, 2009 09:29:10

So would this kill the spawn before or after reaching the memory limit? Sure, it would be handy to have something to prevent your apps from sucking up 16GB of memory, but it would be REALLY nice to have something that frees the memory of big requests after they've completed. Yes, max requests works, but seems like a workaround if this is your goal.

[FooBarWidget]

From jamesrtkay on April 28, 2009 13:11:44

This limit causes the process to die as it attempts to exceed its maximum permitted size; the request would fail. As such it is a last resort (but sometimes things don't work as you would want them to and you need a fallback).

At the end of our request processing logic we have already patched Passenger's abstract_request_handler to ensure that all the garbage collection is concentrated in one place: the alternative is to leave it to ruby's defaults which often cause multiple garbage collection scans per request: very inefficient. Before garbage collecting we are going to check for large process size and exit when the process is overly large by checking the RSS: @Woody, is this what you are referring to?

There is really no substitute for asking the OS to enforce hard limits for when your code is not behaving. If you don't and you are on Linux then you are at the mercy of the OOM Killer. @nmeans, although Passenger's code is not defective, the absence of this function does mean that the solution as a whole is less robust than it could be.

[FooBarWidget]

From luke%lukeludwig.com@gtempaccount.com on June 23, 2009 07:21:43

This feature, restarting app instances when their memory grows beyond a max limit, would be found useful by many people. For example, our app is hosted at Engine Yard and they have even written a shell script for their customers which we run as a cron and which uses passenger-memory-stats to determine if an instance has consumed more than a given memory limit and kills the instance if it has. This works, but it would be nice if it was built-in.

[FooBarWidget]

From frankske on September 04, 2009 08:38:17

Would be real handy, esp for passenger nginx. The PassengerNginxHelperServer is not a subprocess of nginx, so the "normal" monit stuff won't help.

[FooBarWidget]

From John.R.Mandel on October 23, 2009 13:00:10

This is exactly what we need to defend against errors in the system that cause the passenger instance to grow wildly out of control. Right now on EC2 when you run out of memory the instance freezes so this is a system down situation when such a bug (albeit ours) occurs.

[FooBarWidget]

From weibel on March 22, 2010 04:37:24

Has the PassengerMemLimit been inuded in recent releases of Passenger? I would find it really useful as I have an app that is suspectible to consume all RAM under certain unfavourable conditions over which I have no control.

[FooBarWidget]

From will.bryant on August 08, 2010 02:37:42

This code looks like it takes the right approach to me, and I'd like to see it added to Passenger - I've now seen several cases where a client's bad code created gigabytes of Ruby objects and slowed their VM to a crawl and so affecting other requests being served; having the offending code die after bloating past 1GB would be far preferable, and the downsides very few (since such pages will always have timed out for the HTTP client by this point anyway, we might as well let it segfault and die).

weibel, I had a brief look at the current passenger head and I don't see anything like this in it.

BTW - Tim Copeland blogged the following in June 2009, with his update to this patch but also talking about making a custom build using it: http://getindi.com/articles/tom/updating-the-passengermemlimit-patch (woody.peterson, I think you may misunderstand the intention of this patch, it is not a current memory check that Passenger would run before or after spawn, it is a limit applied to the host OS that will cause the process to fault at the point where it tries to exceed the memory limits - ie. it is a damage mitigation tool. The memory used by big requests will always be freed after the big request completes unless you've done something silly (like hold on to references from live objects, defeating garbage collection).)

So it seems we're waiting to hear if this will be considered for future versions of Passenger. If not I guess we could try and write this as a standalone Ruby gem, which would mean losing the ability to change the limit at runtime using the Passenger configuration mechanism but would gain the ability to use it for non-HTTP jobs too, and would potentially allow the Ruby code to vary its own limits when required. I might have to give that a try.

[FooBarWidget]

From seph+old@directionless.org on September 30, 2010 05:35:24

+1 from me as well.

We recently switched to passenger from mongrel, and this is a real problem for us. Using mongrel, we had monit configured to kill things if they got out of hand. But I don't see anything as clean for passenger. I'll probably write some kind of shell script to monitor it, but I'm sad to have lost the feature.

For me, max_requests doesn't help, it's not a memory leak over time. Sometimes an app tries to buffer too much data into RAM, and brings everything to a screaming halt.

[FooBarWidget]

From Ivan.Y.Kuznetsov on February 21, 2011 13:32:10

This article describes a workaround for this problem http://www.hskupin.info/2010/06/17/how-to-fix-the-oom-killer-crashe-under-linux/

[FooBarWidget]

From b.m.tenbrinke on January 16, 2012 04:20:08

I've patched passenger-memory-stats to function as a watchdog for this / kill passenger instances with high memory usage: https://github.com/FooBarWidget/passenger/pull/39

[FooBarWidget]

From space.ship.traveller on July 16, 2012 06:23:07

I'm not sure if this helps but I made a script which can be used to periodically check for excessive memory usage: http://www.codeotaku.com/blog/2012-06/passenger-memory-check/index

[FooBarWidget]

From honglilai on October 24, 2012 10:38:16

This feature has been implemented in Phusion Passenger Enterprise.

Status: Fixed
Labels: -Type-Defect Type-Enhancement

[FooBarWidget]

From rogerdpack on June 14, 2013 09:33:34

what is the directive to enable this then?