Pre-processing leaks memory

[pushred]

[pushred]

Here's overnight without any preprocessors running:

[Fauntleroy]

I'm assuming that this is some sort of problem involving dying vms. I'll have to make some test cases and see what's up.

[pushred]

Looks like a preprocessor need not even do anything to leak memory, just merely be referenced from the page. Below is the contents of such a preprocessor and it's memory consumption after several hours (the drop at the end is from a restart):

/* player */
//{ url: insta_vid_list[random].url, username: insta_vid_list[random].username}

[pushred]

2 days running:

[joanniclaborde]

After increasing the VM's memory size to 1GB, the memory usage on Carrie peaked at 557MB after 5 hours. Also, I was never able to experience a memory leak on my machine, after a full day of playing around and trying different configurations.

I think Solidus is just hungry. Each worker can take up to 70mb of memory, from what I could tell. The rest of the system would need to be tested, to see where that memory is used.

[joanniclaborde]

After digging through the code, I can't find any real problem. The memory grows simply because memory is available, V8 doesn't bother releasing it. For example, when I trace the memory usage:

$ NODE_ENV=production node start.js
118.02 after server is started
243.63 after templates are loaded
252.02 after page is rendered
... // many many requests
518.34 after page is rendered

And do the same thing, but I force a garbage collection after the templates are loaded (big drop in memory):

$ NODE_ENV=production node --expose-gc start.js 
118.36 after server is started
243.83 after templates are loaded
115.39 after forcing garbage collection
133.93 after page is rendered
... // many many requests
563.52 after page is rendered

Forcing garbage collection is not really a solution, we should let V8 take care of that. But, we can tell it the maximum memory it can use:

$ NODE_ENV=production node --max_old_space_size=256 start.js 
118.05 after server is started
191.21 after templates are loaded
192.64 after page is rendered
... // many many requests
277.50 after page is rendered

I'll send an email to Modulus about this. It doesn't make sense to start a Node process on a 396mb machine, without changing the default max_old_space_size, which is much higher.