search

wandenberg / nginx-push-stream-module

A pure stream http push technology for your Nginx setup. Comet made easy and really scalable.
Other
2.22k stars 295 forks source link

Possible memory leak #65

Closed tomtaylor closed 11 years ago

tomtaylor commented 11 years ago

I'm getting a memory leak in nginx, which I think is probably caused by the nginx-push-stream-module. I'm using 82e685e31c567c8d747c1aff513e39249d00f9ec on nginx 1.2.7.

After about 15-20 days, the error log starts producing "2013/04/11 15:15:02 [crit] 14628#0: ngx_slab_alloc() failed: no memory" and every 1 in 3-4 requests to publish messages into channels starts bypassing the push stream location directive and hitting our main web application. When we restart nginx, everything is fine again.

Our usage of push-stream-module is probably a little atypical. We have lots of channels which are only published to 2-3 times, and often never received on. We keep one message in backlog and that's always requested if a client connects to a channel. The message TTL is 5 mins. We're allocating 128MB for memory space, which should be plenty for our usage.

We've since restarted the production application this was happening on, so I can't get better diagnostics out of it. Hopefully this might give a bit of a starting place, but let me know if I can provide more detail.

Downchuck commented 11 years ago

The system tap toolkit might be helpful in analyzing the situation.

tomtaylor commented 11 years ago

Thanks, but unfortunately I don't think we'll be able to use that on Linux 2.6, which is what our production box is running.

Downchuck commented 11 years ago

You may be able to use a fresh Linux box in a VM to hunt for a leak.

On Apr 13, 2013, at 12:58 PM, Tom Taylor notifications@github.com wrote:

Thanks, but unfortunately I don't think we'll be able to use that on Linux 2.6, which is what our production box is running.

— Reply to this email directly or view it on GitHub.

tomtaylor commented 11 years ago

I've deployed 0.3.5 and set the new channel timeout option to the same as the message timeout (5 mins). I'll report back in a couple of weeks if it fixes things, or not.

Were empty channels being removed from memory before? If not, that might go some way to explaining why we hit our memory limit. How many bytes does an empty channel use?

wandenberg commented 11 years ago

Hi Tom,

setting channel timeout to 5 min, without changing any other value, may turn things worst. The default timeout is 30 s, if your turn this value to 5 min an inactive channel will stay a longer time on memory and you will get "no memory" message sooner.

The minimun memory size used to the main module structures are here memory-usage.

Remember that, 128Mb may be ok to a normal usage of your application, but not for a peak time.

Try to monitor the behavior of your application, and adjust the shared memory size to support peak time, you can use the information on channels-stats location to know how many messages/channels/subscribers are on shared memory, in use and on trash waiting to be released.

wandenberg commented 11 years ago

Hi Tom, any news about this problem?

wandenberg commented 11 years ago

Hi,

recently we map that this may be a known issue on nginx core. Nginx is prepared to work well with blocks of ngx_pagesize, generally 4kb. I sent a patch to nginx-devel list as you can see on the links bellow.

http://forum.nginx.org/read.php?29,240420,240420#msg-240420

https://gist.github.com/wandenberg/6096183