search

TryGhost / Ghost

Independent technology for modern publishing, memberships, subscriptions and newsletters.
https://ghost.org
MIT License
47.26k stars 10.31k forks source link

Challenge: Testing / CI for memory usage #1914

Closed ErisDS closed 7 years ago

ErisDS commented 10 years ago

I'd love to see some sort of reporting on how much memory Ghost uses as part of our build / CI.

Even though this information is readily available via process.memoryUsage(), all of the information / tools around seem to be focused on instrumenting the app and detecting memory leaks.

Although if we have memory leaks, we should find them, this is not what I'm really interested in. I'm more interested in how much memory the app is using overall on average, so we can keep an eye on usage as we grow the application.

Every additional dependency, all the modules we have internally, all the data we store in memory etc all has an effect on memory usage, and it would be nice to track this as we add new features.

ErisDS commented 10 years ago

accidental close :/

cgiffard commented 10 years ago

Underway on my fork. :)

ErisDS commented 10 years ago

Excited about this. I think this will introduce the need for nightly builds - which is something I've been thinking about having again for a while.

ErisDS commented 10 years ago

@cgiffard I've assigned this one to you - would love an update!

cgiffard commented 10 years ago

I'm making some core changes in SteamShovel to generate better statistics. It's currently working, but I can make it better.

JohnONolan commented 10 years ago

Relevant: I saw someone report a 30% increase in memory usage after upgrading to 0.5 on Twitter the other day

Swaagie commented 9 years ago

Unsure how much has been done with this, but the nodejitsu blog just stopped working on a 1024mb drone. 'Stopped' as in the admin reporting a ENOMEM. It basically consumed the larger portion of all that memory.

jaswilli commented 9 years ago

Hi @Swaagie,

It looks like that blog is still running 0.5.0? I don't think there were any identified leaks specific to that version, however, there is/was a known issue with the New Relic agent leaking memory (https://github.com/TryGhost/Ghost/issues/3980#issuecomment-54979472), if you happen to be using it...

If not, I'd recommend upgrading to 0.5.8 and seeing if the memory usage remains the same. If it does we'd love a heads-up :smile:

megastef commented 9 years ago

Hi, I monitor a ghost blog using SPM. The screenshot in the blog is actually from a Ghost Server. http://blog.sematext.com/2015/03/30/nodejs-iojs-monitoring/ Live data is available in "Node.js Demo" https://apps.sematext.com/demo npm/github link inhere: https://www.npmjs.com/package/spm-agent-nodejs

Let me know if I could help, we used it e.g. to compare our agents memory footprint on io & node. We found out that on 0.12 or io.js it can take hours until the memory gets back to low level - if you just watch a short time, while the server has not much to do you might get the impression that memory is always growing. Here is a chart over 2 days in such a situation https://apps.sematext.com/spm-reports/s/LmdiChz3UR

ErisDS commented 9 years ago

Hi @megastef I looked into this when you first posted it and it looks really interesting. I also saw your tweet the other day and wondered if you had any insights into how to improve that behaviour?

megastef commented 9 years ago

Hi - about the numbers it would need a clean test on recent versions e.g. 0.12 / ghost 0.6 - the chart from the tweet is a ghost blog I'm running on my web hoster and it's not really up to date because i'm happy with it and it's running well for several month. I think the node version is 0.10.36, using ghost 0.5.x (an early one). After adding monitoring I used 'siege' to put constantly traffic to it ~10 req/s - and just wondered how GC takes ~8% of time. So something keeps the GC busy - I guess a lot of temp. objects created, perhaps you can cache more things, ... http://stackoverflow.com/questions/18364175/best-practices-for-reducing-garbage-collector-activity-in-javascript and http://stackoverflow.com/questions/18800440/javascript-and-garbage-collection might help you.

In general I see in most node 0.10.x versions growing memory - the recent 0.12.x versions and latest io.js release looks fine in this regards - but sometimes on low load it takes hours until the low memory is reached again - so when watching memory for a short time you might get the impression of memory leaks - but this might not be the case, be patient. For example here all running the same program on different node/io versions:

I suggest to setup latest ghost on differnt node/io versions and see if it still behaves like the one in my tweet. Just drop me an e-mail and we can arrange e.g. a call to see what else could be done (e.g. add custom metrics or logging to Logsene to correlate logs with metrics in SPM).

megastef commented 9 years ago

Every additional dependency, all the modules we have internally, all the data we store in memory etc all has an effect on memory usage, and it would be nice to track this as we add new features.

Just a few more ideas on the original issue. You could use daily e-mail reports as PDF/Image, create it on demand or you can save metrics in a shareable link - in that way you could compare memory of build/test run. Like the link I used in the tweet - it's fixed to a specific time, and secured to share. It could be useful to create the reports at the end of a test cycle. A sample of such a report

ErisDS commented 9 years ago

I think the node version is 0.10.36, using ghost 0.5.x (an early one).

Are all of these charts from such an old version? Almost every line of Ghost has changed in that time!

megastef commented 9 years ago

Hi, I had quick look into charts for Ghost 0.6.2 running on node 0.12.2 / ubuntu / digitalocean: It's much better ~1-2 seconds GC time per minute when using ~10 req/s. To reach again 5-6s and high number of GC runs (~300/s) it needs now a 10 times higher request rate! (~100 req/s).

HTTP req: https://apps.sematext.com/spm-reports/s/bKylFGuFAX HTTP res. times: https://apps.sematext.com/spm-reports/s/0yCd9TXARp GC Times: https://apps.sematext.com/spm-reports/s/n6ZIC2hi5z GC Runs: https://apps.sematext.com/spm-reports/s/ReJHSCKGGz Actually it looks that a lot of memory is released (which is good) GC released memory: https://apps.sematext.com/spm-reports/s/5CVv3mmkLV Memory usage: https://apps.sematext.com/spm-reports/s/36o2iMCTTa Event Loop latencies: https://apps.sematext.com/spm-reports/s/j2y1y4Tt8A CPU usage: https://apps.sematext.com/spm-reports/s/rcbF0qbJvX

(it would be easier to share the whole app, via e-mail .. But you can try it yourself, use a free account and observe it - let me know if you need help to get started).

Some news regarding the original issue of continuous measurements during tests: We just implemented the API feature to trigger reports via the API (it's called 'subscription' in the UI). This means the scenario to run benchmark and trigger then e-mails with the charts for documentation is now possible using SPM HTTP JSON/API. We will update the API documentation soon.

ErisDS commented 7 years ago

Challenge was never accepted 😂

We have various monitoring available to us on Ghost(Pro). If anyone ever has a nice idea for a solution to this let us know, we're interested. In the meantime gonna close!