search

TwP / logging

A flexible logging library for use in Ruby programs based on the design of Java's log4j library.
https://rubygems.org/gems/logging
MIT License
529 stars 100 forks source link

Deleted files kept open #232

Closed lzap closed 2 years ago

lzap commented 3 years ago

Hey,

we have a report from a customer about some deleted and opened files left after SIG1 is used to trigger Logger.reopen.

[12:10:50] [root@bombsat69 ~]# lsof  -n|grep -e PID -e delete|grep proxy.log
ruby       1049       foreman-proxy    9w      REG              253,0      51926  403130400 /var/log/foreman-proxy/proxy.log-20210721 (deleted)
ruby       1049       foreman-proxy   15w      REG              253,0      51926  403130400 /var/log/foreman-proxy/proxy.log-20210721 (deleted)
ruby       1049       foreman-proxy   16w      REG              253,0      51926  403130400 /var/log/foreman-proxy/proxy.log-20210721 (deleted)
ruby       1049       foreman-proxy   17w      REG              253,0      51926  403130400 /var/log/foreman-proxy/proxy.log-20210721 (deleted)

After reading the rolling mechanism, my conclusion is that reopen process is triggered during copy operation, that is during FileUtils.concat call. Or I cannot explain why would the library open a copy of a file. Do you have an idea how this could happen? Have you seen such a problem before? Thanks.

lzap commented 3 years ago

Observation: The customer uses logrotate to send the signal with "compress" option, it is possible that the file is compressed and deleted while it is still copying. I asked them to use "delaycompress" logrotate feature which delays compression of rotated files by one cycle (one day in this case). This should give a lot of time to finish copying.

TwP commented 2 years ago

Which process is rolling the log files in this situation? Is logrotate being used to roll the log files and then signal the ruby process to re-open the files? Or is the logging gem itself configured to perform the log file rotation?

If you could include the logroate configuration and the Logging gem configuration, that would be helpful.

lzap commented 2 years ago

Yeah, I don't remember how this ended up being resolved, but it's been fixed. Thanks!