search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.33k stars 1.05k forks source link

log rotation does not release deleted files / holds disk space #6532

Open jalogisch opened 4 years ago

jalogisch commented 4 years ago

Expected Behavior

The logfile rotation should release the file handle and free disk space.

Current Behavior

When many logs are created within a short time, the filesystem handles and disk space is only release on Graylog restart.

[gl2adm@rsv01 graylog]$ ll /var/log/graylog/
-rw-r--r--. 1 gl2adm gl2adm 79650 Sep 18 17:45 graylog-server-1.log.gz
-rw-r--r--. 1 gl2adm gl2adm 83328 Sep 18 17:50 graylog-server-2.log.gz
-rw-r--r--. 1 gl2adm gl2adm 71821 Sep 18 17:51 graylog-server-3.log.gz
-rw-r--r--. 1 gl2adm gl2adm 76595 Sep 18 17:54 graylog-server-4.log.gz
-rw-r--r--. 1 gl2adm gl2adm 77732 Sep 18 17:59 graylog-server-5.log.gz

[gl2adm@rsv01 ~]$ lsof | grep -E 'graylog.*\(deleted\)'
java 15931 gl2adm 1w REG 253,4 1630832143 275859 /var/log/graylog/graylog-server-5.log (deleted)
java 15931 gl2adm 2w REG 253,4 1630832143 275859 /var/log/graylog/graylog-server-5.log (deleted)
java 15931 3969 gl2adm 1w REG 253,4 1630832143 275859 /var/log/graylog/graylog-server-5.log (deleted)

this might be limited to xfs filesystems, but not sure on that.

Steps to Reproduce (for bugs)

  1. us this log4j.xml
  2. have Graylog rotate the logs very fast (see timestamps above)

Context

When you are in a high log situation it is very likely that the reason is high load, what means you can't restart Graylog to free up the disk space. But Graylog will stop working if the logging partition does no free space available.

Your Environment

[z#842704]

jalogisch commented 4 years ago

this does work without a problem when you have a "normal" log rotation that does not kick in every few minutes.

We need to verify how that behaves when you have high volumen log files.