search

mnutt / hummingbird

Real time web analytics using node.js and web sockets
http://mnutt.github.com/hummingbird
MIT License
2.92k stars 15 forks source link

Memory leak? #50

Closed liblfds closed 11 years ago

liblfds commented 11 years ago

Hi.

I have Hummingbird running with the current node.js (0.8.22). I was using 0.8.16 before, same issue.

We're running on an amazon VM, x1.small, which has a total of 1.7 GB of RAM and 1 GB swap.

The server is somewhat loaded - about 400 packets per second. I've set ulimit -n to 100000 (having adjusted limits.conf to allow this).

The problem is memory use simply rises over time. At 400 packets a second, I have some hours before I'm at 80% memory (it's all held by node.js) and then I restart the server.

I've read node.js has a 1.4 GB heap limit, so presumbly if I wasn't stopping it early, it would soon reach that limit anyway and have problems.

I can't see that node.js is doing anything wrong, so I'm back to thinking this is an issue with Hummingbird, but obviously I'm not seeing here other people reporting this, so I wonder!

mnutt commented 11 years ago

I'm almost 100% sure this is related to the geoip module. I've just updated from 0.4.6 to 0.4.8 and I think it may slow the leak down, but doesn't fully stop it. I'm going to look into either fixing the library or moving to a different one.

I think this hadn't been widely reported because most people have low-traffic sites and I hadn't noticed it in load testing because it requires using a much larger list of IPs than I was using. Now that I'm using all of the IPs in the geoip database it is very clearly leaking.

mnutt commented 11 years ago

I think this should fix it; if you still see it please reopen. The next step would be to try turning off the locations metric and see if the problem persists.

liblfds commented 11 years ago

Thanks, mnutt. I will experiment tomorrow morning and report back.

liblfds commented 11 years ago

For others, the command to install is;

cd [hummingbird dir] npm install geoip-no-city-leak@0.4.9-1

then fix up your lib/metrics/locations.js file (and any copies from it) to be;

-var geoip = require('geoip'); +var geoip = require('geoip-no-city-leak');

mnutt - I've made the change, we're testing now.

liblfds commented 11 years ago

Memory leak is indeed fixed.

Thanks for the prompt and effective product support, mnutt.