xcat2 / xcat-core

Code repo for xCAT core packages
Eclipse Public License 1.0
356 stars 170 forks source link

Do not use copytruncate for xcat log rotation #7425

Closed Obihoernchen closed 4 months ago

Obihoernchen commented 4 months ago

PR https://github.com/xcat2/xcat-core/pull/6510 tried to fix missing logs for goconserver, but also added copytruncate to all xcat logs in /var/log/xcat*.log. This is not needed because these logs are written by rsyslog and xcat itself, not goconserver.

The main rsyslog developer does not recommend to use copytruncate for rsyslog: https://serverfault.com/a/901366

For HA setups with logs on NFS etc. copytruncate can be very slow. A simple move/rename is way faster and cleaner.

I'm keeping delaycompress because I think it's nice to have some more days/weeks of logs uncompressed. Furthermore, some users might be used to it already.

Obihoernchen commented 4 months ago

Yes but this shouldn't be the case here because we do:

 test -f /var/run/rsyslogd.pid && kill -HUP `cat /var/run/rsyslogd.pid 2> /dev/null` 2> /dev/null || true

The official rsyslog-logrotate package is doing the same:

[root@service01 ~]# cat /etc/logrotate.d/rsyslog
/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
{
    missingok
    sharedscripts
    postrotate
        /usr/bin/systemctl -s HUP kill rsyslog.service >/dev/null 2>&1 || true
    endscript
}

I tested the change and for me it's working fine. And before https://github.com/xcat2/xcat-core/pull/6510 there was no issue with xCAT logs either.

But please test on your system, too.

samveen commented 4 months ago

Given that rsyslog honors the HUP signal to close and reopen logs, copytruncate isn't required for it. The postrotate script just needs to ensure that the rsyslogd is correctly signaled about log rotation completion.

Ubuntu 20.04 uses the following:

samveen@prod-1:~$ grep '^VERSION=' /etc/os-release 
VERSION="20.04.6 LTS (Focal Fossa)"
samveen@prod-1:~$ grep postrotate -A2 /etc/logrotate.d/rsyslog 
        postrotate
                /usr/lib/rsyslog/rsyslog-rotate
        endscript
--
        postrotate
                /usr/lib/rsyslog/rsyslog-rotate
        endscript

Ubuntu 22.04 uses the following:

samveen@Ubuntu-2204-jammy-amd64-base ~ # grep '^VERSION=' /etc/os-release; grep -A2 postrotate /etc/logrotate.d/rsyslog 
VERSION="22.04.3 LTS (Jammy Jellyfish)"
        postrotate
                /usr/lib/rsyslog/rsyslog-rotate
        endscript

Debian 12 Bookworm uses the following:

samveen@zero2w:~/workspace/xcat-core $ grep '^VERSION=' /etc/os-release 
VERSION="12 (bookworm)"
samveen@zero2w:~/workspace/xcat-core $ grep -A2 postrotate /etc/logrotate.d/rsyslog 
        postrotate
                /usr/lib/rsyslog/rsyslog-rotate
        endscript

I'd suggest adding the debian/ubuntu check+execute into the postrotate section of xCAT/etc/logrotate.d/xcat , or I can create a PR for this change this is merged.

Obihoernchen commented 4 months ago

There is no /usr/lib/rsyslog/rsyslog-rotate on EL, therefore I would just stick to the current implementation of sending the HUP signal to /var/run/rsyslogd.pid. This should work on most OSs.

samveen commented 4 months ago

@Obihoernchen Yes that script doesn't exist for the RH based distributions. However:

What I am proposing is to add the following to the list in the postrotate above:

        test -x  /usr/lib/rsyslog/rsyslog-rotate && /usr/lib/rsyslog/rsyslog-rotate || true

This fixes a hole in Ubuntu support, as well as increases Debian compatibility. It's also possible that copytruncate was specifically added just because such signalling is missing on Ubuntu, thus leading to the error #6510 was trying to fix.

Obihoernchen commented 4 months ago

@samveen Very good point thank you! Added.

samveen commented 4 months ago

@Obihoernchen Trying to get closet to the tip than the base in Graham's Hierarchy of Disagreement :rofl: