mjy78
commented
3 years ago Experienced this on our production environment overnight as well. Have a max_size of 1G. It looks like logrotate created a copy of the log when it reached this size, but then didn't (or couldn't reset the main log) so the main log just kept growing and logrotate kept creating copies of the log until the file system was out of space.
My settings are as per below:
$ pm2 set pm2-logrotate:max_size 1G
$ pm2 set pm2-logrotate:retain 30
$ pm2 set pm2-logrotate:compress true
$ pm2 set pm2-logrotate:dateFormat YYYY-MM-DD_HH-mm-ss
$ pm2 set pm2-logrotate:workerInterval 300
$ pm2 set pm2-logrotate:rotateInterval 0 0 0 1 * *
$ pm2 set pm2-logrotate:rotateModule true
When the log-rotator hits rotate time it maxes out cpu and memory and creates a log file that completely fills the logs drive (in this case 27Gb). The whole machine grinds to a halt.
I have had to disable it. Even though I have deleted all of the offending log files the 27Gb space is not freed.
log-rotator is process zero in pm2 and I did pm2 stop 0 possibly this has locked the main log file?