Closed Cow258 closed 5 years ago
Hi, did you see and read all of the comments on #955?
I read all about 20-30 times, i tried everything, but the issue still here
Easy to reach 500mb by uploading 16 images and each 6MB
module.exports = {
apps: [{
name : 'novel',
script : './app.js',
watch : false,
//node_args : '--max_semi_space_size=2 --max_old_space_size=128',
env : { 'LD_PRELOAD': '/usr/lib/x86_64-linux-gnu/libjemalloc.so.1', 'NODE_ENV': 'production' },
env_production: { 'LD_PRELOAD': '/usr/lib/x86_64-linux-gnu/libjemalloc.so.1', 'NODE_ENV': 'production' },
wait_ready : true,
listen_timeout: 2000,
kill_timeout : 2000,
exec_mode : 'cluster',
instances : 1
}]
};
I just tried #1041
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 node app.js
Memory usage below 150mb and no any memory leak issue.
But when using pm2 to start app.js, the memory leak issue still here
The LD_PRELOAD
environment variable must be set before the Node process is started, so probably needs to be configured long before pm2 is involved.
Remember: this is not a memory leak, you are seeing the effects of freed, possibly fragmented memory being held by some memory allocators rather than returned to the OS.
I tried LD_PRELOAD
before pm2 since system boot up, but issue still here.
Maybe issue on pm2.
Because I tried
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 node app.js
Memory usage below 150mb and no any memory issue.
Any way, I'm sure LD_PRELOAD
environment variable was set before the Node process is started.
Thanks, it sounds like you'll need to ask the pm2 maintainers about this - good luck!
Finally, I found out the problem!
This issue only issue on pm2 cluster mode.
pm2 fork mode able to load libjemalloc
correctly but cluster.
Some how the pm2 cluster mode unable to load libjemalloc
correctly.
@Cow258 Add 'max_memory_restart' to limit memory?
@Cow258 Add 'max_memory_restart' to limit memory?
if restart when processing image that cause data lose. so far, using pm2 fork mode will not memory leak
@Cow258 Thanks for documenting your investigation, this has helped me with a similar concern. I'm using pm2 in cluster mode and I notice memory start low and climb up to the max available. The frustrating part is not being able to use cluster mode with pm2 in this situation - this only appears to be related to using this sharp
package on a Node.js instance/cluster though.
@Cow258 also - the way the docs for pm2 explain max_memory_restart
option when in cluster mode is that it's graceful reload so you shouldn't have data loss - I'd be curious to know if you experienced data loss when using this option.https://pm2.keymetrics.io/docs/usage/memory-limit/
@Cow258 also - the way the docs for pm2 explain
max_memory_restart
option when in cluster mode is that it's graceful reload so you shouldn't have data loss - I'd be curious to know if you experienced data loss when using this option.https://pm2.keymetrics.io/docs/usage/memory-limit/
You are correct, graceful reload will not have data loss, but if user uploading 100+ images to your server, this may cause the server to fail to restart in time and then run out of memory and then the server will hang out.
I'm using AWS free tier and transfer data to new account each year, so i got a EC2 with 1CPU and 1GB Ram, that mean i only have limited memory. Also it is running about 3 web server of Node.js with using Sharp.js. Now all server is running on PM2 fork mode without any memory leak problem.
Here are screenshot. After 32 days up time, it still below 150MB of RAM usage.
Finally, I will keep asking to PM2 maintainers about this.
https://github.com/Unitech/pm2/issues/4375#issuecomment-652366370
We can use Cluster mode
with LD_PRELOAD
now!!!\
Required to set LD_PRELOAD
environment variable before pm2
start,\
So...\
Add LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1
\
Into /env/environment
\
Also do\
export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1
\
Then\
sudo reboot
The cluster mode will working with LD_PRELOAD=xxxx
,\
Not issue when processing many big size images with sharp.js
.
https://github.com/Unitech/pm2/issues/4375#issuecomment-898658080
Some how I got memory leak again this few week. So I switched to jemalloc 5.2.1 to fix memory leak issue. Here are some tutorial
Refer: https://gist.github.com/diginfo/be7347e6e6c4f05375c51bca90f220e8
sudo apt-get -y install autoconf libxslt-dev xsltproc docbook-xsl
git clone https://github.com/jemalloc/jemalloc.git
cd jemalloc
autoconf
./configure
make dist
sudo make install
You can found jemalloc at /usr/local/lib/libjemalloc.so.2
sudo su
vi /etc/enviroment
# Then add this line
LD_PRELOAD=/usr/local/lib/libjemalloc.so.2
Then
export LD_PRELOAD=/usr/local/lib/libjemalloc.so.2
Then optional reboot sudo reboot
A reminder to those visiting this page, as I see a number of recently-linked issues: this is not a memory leak, you are seeing the effects of freed, possibly fragmented memory being held by some memory allocators rather than returned to the OS.
Make sure you're using the latest version of sharp (v0.29.0 as of writing), which will determine the memory allocator is in use and adjust concurrency accordingly.
https://sharp.pixelplumbing.com/install#linux-memory-allocator
I faced this problem again in a week, and it took me 2 days to solve it After seeing this comment, I got the answer https://github.com/nodejs/help/issues/1518#issuecomment-997221787
Somehow, the LD_PRELOAD
environment variable may not work
You need to modify /etc/ld.so.preload
to solve this problem
# make install version
sudo echo "/usr/local/lib/libjemalloc.so.2" >> /etc/ld.so.preload
# apt-get version
sudo echo "/usr/lib/x86_64-linux-gnu/libjemalloc.so" >> /etc/ld.so.preload
Then restart all node
processes to use jemalloc
for allocation:
# If you are using pm2
pm2 kill
pm2 resurrect
Then check the PID
of your running node
process and plug it into the command below to verify it is using jemalloc
:
ps aux | grep node
cat /proc/<PID>/smaps | grep jemalloc
Update
I faced this problem again in a week, and it took me 2 days to solve it After seeing this comment, I got the answer nodejs/help#1518 (comment)
Somehow, the
LD_PRELOAD
environment variable may not work You need to modify/etc/ld.so.preload
to solve this problem# make install version sudo echo "/usr/local/lib/libjemalloc.so.2" >> /etc/ld.so.preload # apt-get version sudo echo "/usr/lib/x86_64-linux-gnu/libjemalloc.so" >> /etc/ld.so.preload
Then restart all
node
processes to usejemalloc
for allocation:# If you are using pm2 pm2 kill pm2 resurrect
Then check the
PID
of your runningnode
process and plug it into the command below to verify it is usingjemalloc
:ps aux | grep node cat /proc/<PID>/smaps | grep jemalloc
Thank you for the solution, but I still have to warn people here, this solution may cause errors used by other frameworks, when I configure it and to use ”puppeteer“, it reports the error "ProtocolError: Protocol error (Page.navigate): Target closed.", in the process of troubleshooting, I found that it is in conflict with the current solution
@anhiao Thanks for the warning, this problem with puppeteer is unrelated to sharp, and probably also unrelated to jemalloc.
As an aside, there is plenty of discussion and possible solutions on the puppeteer repo e.g. https://github.com/puppeteer/puppeteer/issues/1947 - Make sure you're providing enough RAM for peak memory usage, which may be higher or lower under a different allocator, as well as understanding its configuration relating to memory e.g. --disable-dev-shm-usage
. Further questions about puppeteer should be directed to the puppeteer repo.
What is the output of running
npx envinfo --binaries --languages --system --utilities
?System: OS: Linux 4.15 Ubuntu 18.04.2 LTS (Bionic Beaver) CPU: (1) x64 Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz Memory: 470.87 MB / 983.91 MB Container: Yes Shell: 4.4.20 - /bin/bash
Binaries: Node: 12.6.0 - /usr/bin/node Yarn: 1.15.2 - /usr/bin/yarn npm: 6.9.0 - /usr/bin/npm
Utilities: Git: 2.17.1 - /usr/bin/git
Languages: Bash: 4.4.20 - /bin/bash Perl: 5.26.1 - /usr/bin/perl Python: 2.7.15+ - /usr/bin/python
Instance Infomation AWS t2.micro (1vCPU, 1GB Ram) Ubuntu 18.04 LTS Linux
What are the steps to reproduce?
All of the following code do memory leak
Using Stream
Using Buffer to Buffer
Using File to Buffer
Using File to File
About the issue
The larger the uploaded file, the more serious the memory leak. When i upload 20 images and 5mb for each images, memory usage come to 400MB-500MB.
When I do
process.memoryUsage()
, RSS about 350-450MB and not effect toglobal.gc();
orsharp.cache(false);
.This issue happen on Node.js version 8,10,11,12. I tried on each Node.js version, but the issue still here.
I tried
global.gc();
andsharp.cache(false);
andsharp.concurrency(1);
, also triedLD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1
, but the issue still here.This issue will be more important when server not restart for long time. I tried all above of the codes and no issue on Windows and MacOS. The memory usage keep increase after time and after upload images on Linux.
I tried switch to gm.js and no memory leak issue, but gm.js is too slower then sharp.js.
Please help me to resolve this issue. Thank you.
Here are full code