Open mahalakshme opened 1 month ago
/var/log/syslog
and corresponding rotated files had the largest disk space usage. This was due to systemd journal forwarding logs to syslog.
This forwarding has been temporarily disabled in avni cloud staging, prerelease and prod.
set ForwardToSyslog=no
in /etc/systemd/journald.conf
sudo systemctl restart systemd-journald
application logs -> journal logs -> syslogs were getting bloated due to repeated warning logs from ehcache DefaultSizeOfEngine since release 10.0.x. Log level has been changed to error for this. Syslog itself is being rotated by logrotate but we were writing too fast for the rotation to help.
Deletion of older tomcat access logs being tested with tomcat config server.tomcat.accesslog.max-days
since we are using tomcat 9. If this does not work, will add a cron/logrotate solution for this.
Staging access logs
Marking it as Tech type QA, to ensure we check the server state before the deploy and validate all the fixes after deployment.
This forwarding has been temporarily disabled in avni cloud staging, prerelease and prod. set
ForwardToSyslog=no
in/etc/systemd/journald.conf
sudo systemctl restart systemd-journald
@1t5j0y Should we enable "ForwardToSyslog" on deployment of 10.1 to prod server.?
This forwarding has been temporarily disabled in avni cloud staging, prerelease and prod. set
ForwardToSyslog=no
in/etc/systemd/journald.conf
sudo systemctl restart systemd-journald
@1t5j0y Should we enable "ForwardToSyslog" on deployment of 10.1 to prod server.?
Yes. Updated avniproject/avni-product#1657
Issue:
Not clearing the old access logs is leading to GBs of uncleared old data - which can be cleared regularly to avoid out of disk space issue.
AC:
Clear access-logs older than 3 months from
/var/log/avni_server/log
folder.