search

strongloop / strong-pm

deployer for node applications
http://strong-pm.io
Other
1k stars 71 forks source link

Possible memory leak in pm #257

Closed erikverheij closed 8 years ago

erikverheij commented 9 years ago

Hi there,

The memory usage of my loopback apps increases over times. It seems like there is some memory leak, but I've not yet figured out what the problem is. This issue raised another problem.

My server crashed due to insufficient memory. After this crash the pm has forgotten about all services. It appears that strong-pm/strong-pm.json is completely being reset to the default.

off-topic: Can I restart my apps after x number of requests to prevent running into memory issues?

rmg commented 9 years ago

We are working on a refactor to store this metadata in a sqlite3 db as well as separating the metadata from the continuously growing operations data. This won't help your immediate need, but it should address this problem once it is released.

As for restarting your apps, strong-pm isn't directly aware of the requests that are made to the app. However, if you tracked this number internally and exited when you hit a threshold, strong-pm would restart your app.

sam-github commented 9 years ago

How would we reproduce the .json DB getting reset? Do we need an app that is crashing over and over, or just one that leaks memory, or...? I assume we can't run your app, but if you could give some suggestions on reproducing the issue that would be helpful.

erikverheij commented 9 years ago

Hi Sam,

I had the similar problems while posting this issue. It is hard to reproduce. In my case it happened when the server ran out of memory. I think that crashed the node app and reset the database.

I was hoping someone who knows the code has a clue where to start looking.

I think the best shot would be to simulate the out-of-memory situtation.

Whenever I ran into the problem again I'll try to collect more specific information.

sam-github commented 9 years ago

ok, so if I made an app that just ate memory, deployed it to a pm, and let it run, I might be able to see this? Or would I need to perhaps SIGKILL pm and try to get it during a DB update?

erikverheij commented 9 years ago

On my server PHP script in running each 10mins. Usually this doesn't result in getting out of memory, but as my app memory increased over time at some moment the PHP script seems to have been the last straw to cause the memory shortage.

So in my case this was the flow;

  1. Deploy an app to pm
  2. Let another process eat up all memory
  3. Make a http request to the app (causes a DB-update i guess?).

But since you are planning to implement another DB solution, it seems to me it is not really necessary to fix this issue? As it is occurs only in very specific conditions and it is going to be fixed anyway.

cgole commented 8 years ago

@erikverheij, new versions of pm have fixes that could have addressed these issues. Can you try out a new version and let us know if it works for you.

erikverheij commented 8 years ago

I can't easily reproduce the phenomenon at the moment. Let's assume it has been fixed. If you don't run out of memory there is in fact no problem at all. That said the updated title of this issue does not really reflect the issue. The memory leak may be also be caused by my own app.