Compress older logs after rotation

[rgov]

I was investigating the OpenRVDAS deployment on RV Odyssey and noticed that the disk was full with over 26 GB of log files. A quick gzip brought that down to 4 GB.

It would be nice if there were an option to compress log files once they're rotated out. Or keep n days of uncompressed logs and compress the older ones.

It's still very easy to use gzipped log files since there's the gzcat utility for shell pipelines and Python has the gzip module. You can configure web servers to pass gzipped files through with Content-Encoding: gzip such that the client decompresses them transparently.

[webbpinner]

There might be a utility already installed (on Ubuntu at least) that can help with this: https://www.digitalocean.com/community/tutorials/how-to-manage-logfiles-with-logrotate-on-ubuntu-22-04

[davidpablocohn]

It feels like it should be fairly easy to implement inside the LogfileWriter inside the write_logfile https://github.com/OceanDataTools/openrvdas/blob/master/logger/writers/logfile_writer.py#L179 method. When we open a new logfile, compress the old. The issues I see here are:

1) Is there any way that, having moved on from a current file, the LogfileWriter will ever revisit it to try to write it again? Modulo messing with the system clock, I don't think so, but there is no structural protection against it. I suppose that in the compress_this_logfile() method one would check whether the corresponding gz file already exists, and if so, not compress the current un-zipped one, and issue a loud warning.

2) Is this the right place to add the functionality, vs, as Webb offers, some other process such as logrotate?

Thoughts?

On Mon, Apr 1, 2024 at 10:17 PM Webb Pinner @.***> wrote:

There might be a utility already installed (on Ubuntu at least) that can help with this:

https://www.digitalocean.com/community/tutorials/how-to-manage-logfiles-with-logrotate-on-ubuntu-22-04

— Reply to this email directly, view it on GitHub https://github.com/OceanDataTools/openrvdas/issues/368#issuecomment-2029604197, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFO7V3UBY2HPCH7AYZ7LEKTY3E64JAVCNFSM6AAAAABFQ5LIB2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRZGYYDIMJZG4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[webbpinner]

Was doing work on Revelle and took the logrotate approach for the OpenRVDAS logfiles. Was pretty simply to setup:

1. Created a new logrotate config from openrvdas: /etc/logrotate.d/openrvdas
2. Add the following to the config file:

   /var/log/openrvdas/*.stderr {
       monthly
       compress
       delaycompress
       missingok
       notifempty
       create 644 root root
   }

Will check back in a few weeks to confirm it's working as intended. This VM is running Ubuntu 22.04

[davidpablocohn]

Thanks, Webb!

I'll snooze this until you report back. If all clear, then I'll go ahead and close it out.

On Fri, May 10, 2024 at 3:13 AM Webb Pinner @.***> wrote:

Was doing work on Revelle and took the logrotate approach for the OpenRVDAS logfiles. Was pretty simply to setup:

1. Created a new logrotate config from openrvdas: /etc/logrotate.d/openrvdas
2. Add the following to the config file:

/var/log/openrvdas/*.stderr { monthly compress delaycompress missingok notifempty create 644 root root }

Will check back in a few weeks to confirm it's working as intended. This VM is running Ubuntu 22.04

— Reply to this email directly, view it on GitHub https://github.com/OceanDataTools/openrvdas/issues/368#issuecomment-2104340260, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFO7V3SHWPCJDYPY5HGCFCDZBSMTPAVCNFSM6AAAAABFQ5LIB2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBUGM2DAMRWGA . You are receiving this because you commented.Message ID: @.***>