search

monome / norns

norns is many sound instruments.
http://monome.org
GNU General Public License v3.0
633 stars 147 forks source link

limit linux log sizes #715

Closed catfact closed 5 years ago

catfact commented 5 years ago

not the first time a user has run out of disk space from huge log files

Output from ls /var/log from https://llllllll.co/t/norns-help/14016/972?u=zebra

-rw-r----- 1 root adm 783M Feb 12 06:25 daemon.log.1
-rw-r----- 1 root adm 783M Feb 12 06:25 syslog.1
-rw-r----- 1 root adm 207M Feb 12 22:42 syslog
-rw-r----- 1 root adm 207M Feb 12 22:42 daemon.log
-rw-r----- 1 root adm 948K Feb 12 02:26 kern.log.1
-rw-r----- 1 root adm 707K Feb 12 06:25 messages.1
-rw-r----- 1 root adm 432K Oct 2 06:25 syslog.7.gz

i'm no expert but i thought logrotate was the typical way to deal with this.

(maybe this should be on norns-image)

simonvanderveldt commented 5 years ago

Odd I believe we only use journald which by default doesn't persist. Seems odd that a syslog file would even be created at all.

ngwese commented 5 years ago

Looking at my device I see about 100MB in /var/log. The log files themselves are getting rotated but historic versions are also being retained - in some cases I have upwards of 7 compressed historic logs (across all sub systems)

Looking at /etc/systemd/journald.conf it appears we may be running a default configuration:

/etc/systemd $ more journald.conf 
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
tehn commented 5 years ago

fixed. enabled:

SystemMaxUse=100
SystemMaxFiles=1

https://github.com/monome/norns-image/commit/a13bfec803d87f3151e5c9f51dfd8ef4627b49b6

artfwo commented 5 years ago

@tehn is that file really stored in /dev/systemd/?

tehn commented 5 years ago

yep!

192.168.1.27 ~/norns $ ls /etc/systemd/
journald.conf  network        system       timesyncd.conf  user.conf
logind.conf    resolved.conf  system.conf  user
artfwo commented 5 years ago

but that's /etc. in https://github.com/monome/norns-image/commit/a13bfec803d87f3151e5c9f51dfd8ef4627b49b6 you add command to put it in /dev.

tehn commented 5 years ago

thank you for spotting this :)

simonvanderveldt commented 5 years ago

I'm still unsure where the files in /var/log are coming from, default journald config is to not persist anything and don't forward to syslog. It seems like a syslog daemon is running, but that would mean the Storage option of journald needs to be set somewhere :confused: The options you've changed @tehn don't relate to syslog so probably won't make a difference for the log files show in the issue's description.

tehn commented 5 years ago

@simonvanderveldt any further suggestions what to change?

simonvanderveldt commented 5 years ago

@tehn I'll have a look this weekend

simonvanderveldt commented 5 years ago

journald

It seems like there's only 1 journald config file, the other locations listed in the docs https://www.freedesktop.org/software/systemd/man/journald.conf.html are empty:

cat /etc/systemd/journald.conf 
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg

All settings are commented out, unfortunately it seems like there's no way to let journald/journalctl print which settings it's currently using :(

Assuming the docs are correct the journal itself isn't being persisted since Storage=auto and /var/log/journal doesn't exist

ls -ahl /var/log/
total 20M
drwxr-xr-x  4 root root 4.0K May 24 18:24 .
drwxr-xr-x 11 root root 4.0K May 14 23:27 ..
-rw-r--r--  1 root root 4.1K May 23 22:50 alternatives.log
drwxr-xr-x  2 root root 4.0K May 24 18:13 apt
-rw-r-----  1 root adm  821K Sep  6 21:35 auth.log
-rw-r--r--  1 root root 3.8K May 14 23:22 boot.log
-rw-r--r--  1 root root    0 Apr 18 08:08 bootstrap.log
-rw-------  1 root utmp 3.8K Sep  6 21:35 btmp
-rw-r-----  1 root adm  7.1M Sep  6 21:36 daemon.log
-rw-r-----  1 root adm   98K Sep  6 21:34 debug
-rw-r--r--  1 root root 151K Sep  6 21:22 dpkg.log
-rw-r--r--  1 root root 2.6K May 15 00:17 faillog
-rw-r-----  1 root adm  3.4M Sep  6 21:48 kern.log
-rw-rw-r--  1 root utmp 286K Sep  6 21:35 lastlog
-rw-r-----  1 root adm  2.4M Sep  6 21:34 messages
drwxr-x---  2 root adm  4.0K Mar  5  2018 samba
-rw-r-----  1 root adm  5.5M Sep  6 21:48 syslog
-rw-r-----  1 root adm   72K Sep  6 21:34 user.log
-rw-rw-r--  1 root utmp 455K Sep  6 21:36 wtmp

The changes made by @tehn are still a smart thing to do in case someone were to have norns running for a long period of time the journal might become rather large, so putting a cap on that is a good move :)

syslog

But the initial issue was with log files created by syslog. We now know there's no integration from journald to syslog but for some reason syslog is also being ran independently of journald, no clue why raspbian went with this:

systemctl list-units | grep -i syslog
rsyslog.service                                                                                 loaded active running   System Logging Service                                                                   
syslog.socket                                                                                   loaded active running   Syslog Socket

Maybe we can just disable syslog? Anyone know why it's still there despite journald also being used?

ngwese commented 5 years ago

Investigation of /etc/rsyslogd.conf might be warranted.

https://linux.die.net/man/5/rsyslog.conf

The section on rules and output channels seem like they might offer the control we’re after - that is assuming the rsyslogd is being used on norns (I’m away from a device at the moment)

tehn commented 5 years ago

trying a few things.

for /etc/logrotate.conf

daily
rotate 1
size 1M

for /etc/logrotate.d/rsyslog

/var/log/syslog
{
    rotate 1
    size 1M
...

for /etc/rsyslog.conf under RULES

$outchannel mysyslog,/var/log/syslog,1048576
*.*;auth,authpriv.none  :omfile:$mysyslog
#*.*;auth,authpriv.none     -/var/log/syslog

hopefully this caps everything at 1M

tehn commented 5 years ago

attempted fix https://github.com/monome/norns-image/pull/61

tehn commented 5 years ago

closing this. reopen if we see problems.