Logrotate filled up all disk space (60Gb) in a single day

[christopheSeeka]

Hello,

Few days ago, my webapp stopped work, i checked on the server and it was stopped in pm2 and didn't want restart. I figured that the disk was full (25Gb) so i took an upgrade to 80Gb and installed logrotate but today, 2 days after, the disk is full again.

I did a flush logs command, which did bring it from 80Gb to 61Gb. Then i figured that the .pm2/logs folder was 57Gb and i see 150 log files like "sign-out__2021-02-13_05-36-03.log" for the app for a single day, how can i fix that?

Here my logrotate default config:

"pm2-logrotate": { "max_size": "10M", "retain": "30", "compress": false, "dateFormat": "YYYY-MM-DD_HH-mm-ss", "workerInterval": "30", "rotateInterval": "0 0 *", "rotateModule": true }, "module-db-v2": { "pm2-logrotate": {} } }

If i understand it well it shouldn keep more than 30 of 10Mb max so 57Gb is not normal. Deleted all sign-out__2021-02-13_05-36-*.log files and down to 8Gb I stopped logrotate for now and its ok but would like to know how to solve this issue

Thank you in advance

[lindev0915]

This is not issue. Your configuration is not prefer.

pm2 set pm2-logrotate:retain 50

It will keep only 50 logfiles, If the 51st file will try to perform the logrtate then it will delete the olf log file.

pm2 set pm2-logrotate:size 300M

Once the log file reached the size of 300M then the pm2-logrotate will work

You can avoid disk full issue by using these two configurations

[christopheSeeka]

Thank you very much for the answer, but isn't the config i shared saying 10Mb per log file and retain 30 max already?

"max_size": "10M", "retain": "30",

How this ended filling out the entire disk in one day?

[matthewmayer]

i had a similar issue. New files created every 30 seconds of increasing size:

-rw-rw-r-- 1 ec2-user ec2-user  136M Mar 19 06:39 out-0__2021-03-19_06-09-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user  117M Mar 19 06:39 out-0__2021-03-19_06-09-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   98M Mar 19 06:39 out-0__2021-03-19_06-10-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   90M Mar 19 06:39 out-0__2021-03-19_06-10-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   85M Mar 19 06:39 out-0__2021-03-19_06-11-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   81M Mar 19 06:39 out-0__2021-03-19_06-11-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   77M Mar 19 06:39 out-0__2021-03-19_06-12-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   75M Mar 19 06:39 out-0__2021-03-19_06-12-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   73M Mar 19 06:39 out-0__2021-03-19_06-13-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   71M Mar 19 06:39 out-0__2021-03-19_06-13-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   67M Mar 19 06:39 out-0__2021-03-19_06-14-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   63M Mar 19 06:39 out-0__2021-03-19_06-14-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   60M Mar 19 06:39 out-0__2021-03-19_06-15-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   57M Mar 19 06:39 out-0__2021-03-19_06-15-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   55M Mar 19 06:39 out-0__2021-03-19_06-16-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   52M Mar 19 06:39 out-0__2021-03-19_06-16-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   50M Mar 19 06:39 out-0__2021-03-19_06-17-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   48M Mar 19 06:39 out-0__2021-03-19_06-17-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   45M Mar 19 06:39 out-0__2021-03-19_06-18-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   43M Mar 19 06:39 out-0__2021-03-19_06-18-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   41M Mar 19 06:39 out-0__2021-03-19_06-19-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   39M Mar 19 06:39 out-0__2021-03-19_06-19-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   38M Mar 19 06:39 out-0__2021-03-19_06-20-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   36M Mar 19 06:39 out-0__2021-03-19_06-20-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   35M Mar 19 06:39 out-0__2021-03-19_06-21-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   33M Mar 19 06:39 out-0__2021-03-19_06-21-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   31M Mar 19 06:39 out-0__2021-03-19_06-22-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   30M Mar 19 06:39 out-0__2021-03-19_06-22-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   28M Mar 19 06:39 out-0__2021-03-19_06-23-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   27M Mar 19 06:39 out-0__2021-03-19_06-23-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   25M Mar 19 06:39 out-0__2021-03-19_06-24-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   24M Mar 19 06:39 out-0__2021-03-19_06-24-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   23M Mar 19 06:39 out-0__2021-03-19_06-25-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   22M Mar 19 06:39 out-0__2021-03-19_06-25-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   21M Mar 19 06:39 out-0__2021-03-19_06-26-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   20M Mar 19 06:39 out-0__2021-03-19_06-26-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   19M Mar 19 06:39 out-0__2021-03-19_06-27-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   18M Mar 19 06:39 out-0__2021-03-19_06-27-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   17M Mar 19 06:39 out-0__2021-03-19_06-28-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   16M Mar 19 06:39 out-0__2021-03-19_06-28-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   16M Mar 19 06:39 out-0__2021-03-19_06-29-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   15M Mar 19 06:39 out-0__2021-03-19_06-29-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   14M Mar 19 06:39 out-0__2021-03-19_06-30-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   13M Mar 19 06:39 out-0__2021-03-19_06-30-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   12M Mar 19 06:39 out-0__2021-03-19_06-31-07.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   11M Mar 19 06:39 out-0__2021-03-19_06-31-37.log.gz
-rw-rw-r-- 1 ec2-user ec2-user   11M Mar 19 06:39 out-0__2021-03-19_06-32-07.log.gz
-rw-rw-r-- 1 ec2-user ec2-user  9.3M Mar 19 06:39 out-0__2021-03-19_06-32-37.log.gz
-rw-rw-r-- 1 ec2-user ec2-user  8.3M Mar 19 06:39 out-0__2021-03-19_06-33-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user  7.4M Mar 19 06:39 out-0__2021-03-19_06-33-37.log.gz
-rw-rw-r-- 1 ec2-user ec2-user  6.7M Mar 19 06:39 out-0__2021-03-19_06-34-07.log.gz
-rw-rw-r-- 1 ec2-user ec2-user  6.2M Mar 19 06:39 out-0__2021-03-19_06-34-37.log.gz
-rw-rw-r-- 1 ec2-user ec2-user  5.5M Mar 19 06:39 out-0__2021-03-19_06-35-07.log.gz
-rw-rw-r-- 1 ec2-user ec2-user  4.8M Mar 19 06:39 out-0__2021-03-19_06-35-37.log.gz
-rw-rw-r-- 1 ec2-user ec2-user  4.1M Mar 19 06:39 out-0__2021-03-19_06-36-07.log.gz
-rw-rw-r-- 1 ec2-user ec2-user  3.3M Mar 19 06:39 out-0__2021-03-19_06-36-37.log.gz
-rw-rw-r-- 1 ec2-user ec2-user  2.6M Mar 19 06:39 out-0__2021-03-19_06-37-07.log.gz
-rw-rw-r-- 1 ec2-user ec2-user  2.0M Mar 19 06:39 out-0__2021-03-19_06-37-37.log.gz
-rw-rw-r-- 1 ec2-user ec2-user  1.5M Mar 19 06:39 out-0__2021-03-19_06-38-07.log.gz
-rw-rw-r-- 1 ec2-user ec2-user 1008K Mar 19 06:39 out-0__2021-03-19_06-38-37.log.gz
-rw-rw-r-- 1 ec2-user ec2-user  500K Mar 19 06:39 out-0__2021-03-19_06-39-07.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:39 out-0__2021-03-19_06-39-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:40 out-0__2021-03-19_06-40-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:40 out-0__2021-03-19_06-40-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:41 out-0__2021-03-19_06-41-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:41 out-0__2021-03-19_06-41-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:42 out-0__2021-03-19_06-42-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:42 out-0__2021-03-19_06-42-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:43 out-0__2021-03-19_06-43-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:43 out-0__2021-03-19_06-43-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user  860K Mar 19 06:44 out-0__2021-03-19_06-44-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:44 out-0__2021-03-19_06-44-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:45 out-0__2021-03-19_06-45-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:45 out-0__2021-03-19_06-45-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:46 out-0__2021-03-19_06-46-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:46 out-0__2021-03-19_06-46-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:47 out-0__2021-03-19_06-47-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:47 out-0__2021-03-19_06-47-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:48 out-0__2021-03-19_06-48-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:48 out-0__2021-03-19_06-48-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:49 out-0__2021-03-19_06-49-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:49 out-0__2021-03-19_06-49-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:50 out-0__2021-03-19_06-50-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:50 out-0__2021-03-19_06-50-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:51 out-0__2021-03-19_06-51-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:51 out-0__2021-03-19_06-51-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:52 out-0__2021-03-19_06-52-06.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:52 out-0__2021-03-19_06-52-36.log.gz
-rw-rw-r-- 1 ec2-user ec2-user     0 Mar 19 06:53 out-0__2021-03-19_06-53-06.log.gz

[pubmikeb]

I have a similar issue (https://github.com/keymetrics/pm2-logrotate/issues/174) with a configuration pm2 set pm2-logrotate:max_size 1K.

It looks like max_size is broken.

[jaylee79]

I had the same issue; crashed my system after logs exploding and taking up 100% of disk space.