Open longG4 opened 1 year ago
cc @nodejs/performance PTAL
@longG4 if your business has such a tight dependency on Node.js, I recommend you to join the team that maintains it. In this way you can analyze and fix those situations when they arises. We are always looking for more companies that are willing to contribute back.
So many things has changed from
Node.js v14 and v18, and some of those could be controlled via ./configure
options. I'd recommend you to explore them.
@mcollina the ./configure options. you mean the configure.py in the source directory?
You can also try bisecting (e.g., just by testing various Node.js versions) to find out if any particular release caused the increase in memory usage. There are thousands of changes between Node.js 14 and Node.js 18, so we cannot possibly say which of them contribute to this.
For us, we went from Node 16 to Node 18 and saw a 30% increase in memory usage across all apps. We haven't done a git bisect yet given that that would require us to build every commit of Node to find the exact commit that introduced the regression. It's a shame that upgrading to be on LTS causes us to have to use more memory.
What information or profiling would be helpful to maintainers to help narrow this down? We'd love to work with the community to find the root cause if possible.
For us, we went from Node 16 to Node 18 and saw a 30% increase in memory usage across all apps. We haven't done a git bisect yet given that that would require us to build every commit of Node to find the exact commit that introduced the regression. It's a shame that upgrading to be on LTS causes us to have to use more memory.
What information or profiling would be helpful to maintainers to help narrow this down? We'd love to work with the community to find the root cause if possible.
Was it noticed from the major update or some semver-minors? You can check it by comparing v18.0.0 against v16.15.0
For us we went from 16.18 to 18.12 and that's when we noticed it. We did not attempt any versions in between. From 18.12 to 18.18.2 we did not see the memory usage change. So the increase was between majors. Granted, we didn't test any of version 17 when we made the jump. I know that's less than helpful, but given we are in a larger organization coordinating a node upgrade is a bit costly so it's hard to jump between versions at the moment. I can maybe see if I have spare cycles to show a meaningful reproduction publicly, but I make no guarantees.
To answer your question, it was noticed in the major jump
I did not put much attention into it, but I can concur here. Migrated from Node 16 to Node 18 and started to see my Google Cloud Functions getting OOM errors, so I've started digging and found this thread.
So just to collate what was written here:
16.18 -> 20 seems to result in RSS increase 14.21.3 -> 20 seems to result in RSS increase
I'm not sure which patch version, so I've just gone with whatever nvm install 16.18
does.
The original code didn't quite work from copy -> paste and doesn't lend itself to analysis IMO
Then I wrote a little script to test an arbitrary node version and run the index.js script...
kill -15 $(pgrep node); rm -f $(node --version).log && for run in {1..100}; do; node index.js >> $(node --version).log & sleep 1 && kill -15 $(pgrep node) >/dev/null; done;
NOTE This script is greedy and lazy, and should be run on an isolated machine, not one with other nodeJS instances that might be killed. I did switch from sigkill to sigterm to try to be a bit more polite.
I then after an initial npm init
and npm install express
in the lowest version of node, just used nvm use {version}
and then ran the above script.
I'm honestly not bothered by increased RAM for newer features of Node. While I suspect some of this can be compiled away; I'd likely not use Node for something where 128MB ram or less wasn't available. None of the examples even take up 64MB RAM, and there is a lot of overhead to do cool work in that memory set. I think the trade-off for NodeJS is good, but also wonder if profile guided optimization, or something of that direction (including custom compiled nodejs) might be more your bag at that small size of RAM
Here are the outputs for lts versions:
lts/fermium -> v14.21.3
lts/gallium -> v16.20.2
lts/hydrogen -> v18.20.4
lts/iron -> v20.17.0
Also I just wrote another script because as I looked at the results, I thought. This isn't NodeJS alone.
Here's a 3-liner
const process = require('process');
const processMemory = process.memoryUsage();
console.info(JSON.stringify(processMemory));
Same node versions
NOTE: 16.18 is also in there, as it was in the last set of logs because it's listed in this thread.
Less than 32MB is the summary across all nodeJS version. 🎩 to the team, who really are doing an amazing job at maintaining and creating such a powerful expressive tool, which I often tell people is the most successful C++ API
Details
Our business process has an upper limit baseline requirement,currently switching to a new version of node.js results in an increase in memory.node version v18.14.1 vs v14.21.3, the memory of the same sample code increased by about 10mb. Why did this happen, and what factors?
Node.js version
18.14.1
Example code
The sample code indexTest.js as follows:
With express 4.18.1 we use the command: ./node indexTest.js we can see memory performance information is as follows: node v14.21.3
node v18.14.1
Operating system
Ubuntu 18.04.5 kernel version is 5.10.16.3
Scope
other
Module and version
Not applicable.