search

metwork-framework / mfext

metwork/mfext module
http://metwork-framework.org/mfext
BSD 3-Clause "New" or "Revised" License
4 stars 5 forks source link

Plugin's log saturation #1996

Open matthieumarrast opened 1 week ago

matthieumarrast commented 1 week ago

We temporary set the log level to "DEBUG" in our mfdata module, in production environment, in order to help us solving a critical issue. But one plugin stopped functionning because the log files reached the limit (100M/5 files)

-rw-r--r-- 1 mfdata metwork 100000000 19 nov.  21:53 step_<plugin>_main.log
-rw-r--r-- 1 mfdata metwork  94079566 19 nov.  20:38 step_<plugin>_main.log.20241119203825
-rw-r--r-- 1 mfdata metwork  94129569 19 nov.  20:58 step_<plugin>_main.log.20241119205820
-rw-r--r-- 1 mfdata metwork  94177526 19 nov.  21:18 step_<plugin>_main.log.20241119211815
-rw-r--r-- 1 mfdata metwork  95796003 19 nov.  21:33 step_<plugin>_main.log.20241119213342
thebaptiste commented 1 week ago

Is it a log_proxy issue ? Normally, in this case, a new log file is created and the oldest one is removed (see log_proxy.c and (I think) the plugin should not stop, unless there is another problem (FS full ?). We should try to reproduce your problem with lower limits.

Recently we had an issue about log_proxy using large memory in some cases, with this fix that I have taken in log_proxy release 0.7.4 which is not the release used in Metwork 2.2 (0.5.2). But I'm not sure at all this fix is related to your problem and would have change anything to it.

matthieumarrast commented 1 week ago

I just noticed the log stopped to be written in the log file (the one with size) "100000000", the rotation doesn't take effect and the plugin stopped functionning. I have removed the log files, restarted the plugin and everything was back to normal.

To reproduce I think we should try to fill log files quickly ! 500M in less than one hour.