search

wtsi-hgi / hgi-systems

Orchestration and configuration management of hgi systems (using terraform & ansible)
GNU General Public License v3.0
3 stars 3 forks source link

Limit log size on GitLab runners #18

Closed colin-nolan closed 7 years ago

colin-nolan commented 7 years ago

Logs (syslog and kern.log) on GitLab runners are reaching 100% disk, which causes the machine to break.

colin-nolan commented 7 years ago

LogRotate to be installed on all hosts with root.

colin-nolan commented 7 years ago

logrotate is already installed and running on our hosts. The issue is that, on the gitlab runner, syslog and kern.log are exploding the disk before they can be rotated. /var/log/kern.log is currently set to rotate weekly with 4 backups, /var/log/syslog rotates daily with 7 backups (both sets of backups are compressed).

I'll configure logrotate to deal with these logs more frequently. However, there is still an underlying issue here that something is logging way too much - these log files are growing much faster than those on any other host that I've looked at.

colin-nolan commented 7 years ago

Implemented in db7157a828616dafad495f86564b3268239d98e2.

The underlying issue of whatever is causing the excesive logging still remains.