search

scylladb / scylla-machine-image

Apache License 2.0
19 stars 26 forks source link

Add logrotate to our images #502

Closed yaronkaikov closed 7 months ago

yaronkaikov commented 8 months ago

Based on https://github.com/scylladb/siren/issues/7370#issuecomment-1230860375

Refs: https://github.com/scylladb/siren/issues/7370

fruch commented 8 months ago

I thought that rsyslog isn't installed in the images anymore..

mykaul commented 8 months ago

It has nothing to do with rsyslog - journal logs need to be rotated (as is other logs that might be filling the root disk)

syuu1228 commented 7 months ago

I checked https://github.com/scylladb/siren/issues/7370#issuecomment-1230860375, all of these settings are logrotate settings for rsyslog, but we dropped rsyslog and use journald only, so there's nothing to apply on master image. (and journald automatically rotate logs unlike rsyslog)

mykaul commented 7 months ago

I checked scylladb/siren#7370 (comment), all of these settings are logrotate settings for rsyslog, but we dropped rsyslog and use journald only, so there's nothing to apply on master image. (and journald automatically rotate logs unlike rsyslog)

I think we should tweak its settings - it seem to consume a lot of memory, before rotating. It may be not be rotating enough.

syuu1228 commented 7 months ago

I think we should tweak its settings - it seem to consume a lot of memory, before rotating. It may be not be rotating enough.

I tried to reproduce this on ScyllaDB 5.4.3 image, but not reproduced. When I generated very large amount of journal log using scylla --default-log-level TRACE with cassandra-stress, systemd-journald only use 0.6% of system memory (about 360MB). I think we can keep using default settings.

syuu1228 commented 7 months ago

Closing, since there is no issue on the latest version.

mykaul commented 7 months ago

@syuu1228 - making sure - are we rotating logs?