python / psf-salt

PSF infrastructure configuration
MIT License
111 stars 57 forks source link

cdn-logs: Hourly cronjob for logrotate not firing #293

Closed ewdurbin closed 1 year ago

ewdurbin commented 1 year ago

Our hourly logrotate cron job is not working on Ubuntu 20.04

https://github.com/python/psf-salt/blob/d6c01b1241f402a6e202cc988b5aa5abf090c96a/salt/cdn-logs/init.sls#L17-L19

This is due to logrotate preferring the systemd timer (see first if clause below):

ee@cdn-logs:/var/log/fastly$ ls -alhtr /etc/cron.hourly/logrotate 
lrwxrwxrwx 1 root root 25 Jan  3 15:38 /etc/cron.hourly/logrotate -> /etc/cron.daily/logrotate
ee@cdn-logs:/var/log/fastly$ cat /etc/cron.daily/logrotate 
#!/bin/sh

# skip in favour of systemd timer
if [ -d /run/systemd/system ]; then
    exit 0
fi

# this cronjob persists removals (but not purges)
if [ ! -x /usr/sbin/logrotate ]; then
    exit 0
fi

/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit $EXITVALUE

We should update our configuration to change the systemd timer from daily to hourly on this host for logrotate.

Timer is currently configured as follows by default on installation of logrotate:

ee@cdn-logs:/var/log/fastly$ cat /etc/systemd/system/timers.target.wants/logrotate.timer
[Unit]
Description=Daily rotation of log files
Documentation=man:logrotate(8) man:logrotate.conf(5)

[Timer]
OnCalendar=daily
AccuracySec=12h
Persistent=true

[Install]
WantedBy=timers.target

I'm not positive what the best way to go about editing the timer is, but it may be as "simple" as overriding the file with salt and calling systemctl daemon-reload as we do for services.

References:

ewdurbin commented 1 year ago

You may be able to use the ini file state: https://docs.saltproject.io/en/latest/ref/states/all/salt.states.ini_manage.html#salt.states.ini_manage.options_present to edit the value without creating a full template.

ewdurbin commented 1 year ago

confirmed fixed in #294