search

drachtio / drachtio-fsmrf

Drachtio freeswitch-based media resource function -- http://davehorton.github.io/drachtio-fsmrf
MIT License
48 stars 26 forks source link

Memory leak assistance #23

Open WhiteyDude opened 5 years ago

WhiteyDude commented 5 years ago

Hi Dave,

I've noticed a crash in my application the other day:

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Some research shows that the application exceeded the default heap size of 1.5GB. I've since increased the heap size using the v8 --max_old_space_size option and added some monitoring of the heap size in Zabbix. I've noticed a sizeable increase of roughly 8MB per hour since doing so.

Can you point in the right direction for debugging memory allocation within an application using drachtio-srf and drachtio-fsmrf? I suspect the leak to come from the 1200 odd lines of average-at-best code I've written!

Thanks!

davehorton commented 5 years ago

I've had a feeling that I may have a leak in drachtio-fsmrf, because from time to time I have seen warning messages about the number of listeners I've attached to the mrf object, when I thought I had attached only one. I think I've seen this when reconnecting to the media server, but there may be other instances I need to track down.

So I need to look into this as well, as it may well be something on my side. If you wanted to help that would be great.

I think the way I would go about it would be to create a simple B2BUA using drachtio-srf only (no drachtio-fsmrf). Then I would use sipp or the like to do a sustained load test and see if memory keeps growing. If not, we can move on to looking at drachtio-fsmrf; if so I would then troubleshoot further in drachtio-srf.

Then I would create a similar as simple as possible B2BUA incorporating drachtio-fsmrf and load test that. I'd probably create 2 versions -- first, one that allocates a media server object once at startup and continually creates/deletes endpoints on that media server, and then once that was shown to be leak free, a second version that connects to the media server with every incoming call.

WhiteyDude commented 5 years ago

Thanks Dave!

Unfortunately about 10 minutes after writing that, my youngest smashed my Macbook screen, so I'll be out of commission for a day or two. I'm happy to help out however, will see if I can start writing some test code over the next few days! Will report back with how I go!

davehorton commented 5 years ago

can you retest with the latest drachtio-fsmrf@1.5.2 when you get a chance? I did find and fix some memory leaks in that version. Specifically, when the endpoint was destroyed it was still being retained via a reference in the underlying modesl framework. I believe I have fixed that which should eliminate one large source of a memory leak.

FYI, for future reference running node --inspect and then chrome://inspect in your browser was quite useful for detecting leaks, if you are able to identify and run known scenarios in a non-production environment. Some details here:

https://developers.google.com/web/tools/chrome-devtools/memory-problems/heap-snapshots