avniproject / avni-infra

Other
0 stars 0 forks source link

Remove access logs older than 3 months #45

Open mahalakshme opened 1 month ago

mahalakshme commented 1 month ago

Issue:

Not clearing the old access logs is leading to GBs of uncleared old data - which can be cleared regularly to avoid out of disk space issue.

AC:

Clear access-logs older than 3 months from /var/log/avni_server/log folder.

1t5j0y commented 4 weeks ago

/var/log/syslog and corresponding rotated files had the largest disk space usage. This was due to systemd journal forwarding logs to syslog.

This forwarding has been temporarily disabled in avni cloud staging, prerelease and prod. set ForwardToSyslog=no in /etc/systemd/journald.conf sudo systemctl restart systemd-journald

application logs -> journal logs -> syslogs were getting bloated due to repeated warning logs from ehcache DefaultSizeOfEngine since release 10.0.x. Log level has been changed to error for this. Syslog itself is being rotated by logrotate but we were writing too fast for the rotation to help.

Deletion of older tomcat access logs being tested with tomcat config server.tomcat.accesslog.max-days since we are using tomcat 9. If this does not work, will add a cron/logrotate solution for this.

1t5j0y commented 3 weeks ago

Staging access logs

Before: ``` ubuntu@ip-10-20-1-175:/var/log/avni_server/log$ ls access_log.2023-11-15.log access_log.2024-01-04.log access_log.2024-02-23.log access_log.2024-04-13.log access_log.2024-06-02.log access_log.2024-07-22.log access_log.2024-09-10.log access_log.2023-11-16.log access_log.2024-01-05.log access_log.2024-02-24.log access_log.2024-04-14.log access_log.2024-06-03.log access_log.2024-07-23.log access_log.2024-09-11.log access_log.2023-11-17.log access_log.2024-01-06.log access_log.2024-02-25.log access_log.2024-04-15.log access_log.2024-06-04.log access_log.2024-07-24.log access_log.2024-09-12.log access_log.2023-11-18.log access_log.2024-01-07.log access_log.2024-02-26.log access_log.2024-04-16.log access_log.2024-06-05.log access_log.2024-07-25.log access_log.2024-09-13.log access_log.2023-11-19.log access_log.2024-01-08.log access_log.2024-02-27.log access_log.2024-04-17.log access_log.2024-06-06.log access_log.2024-07-26.log access_log.2024-09-14.log access_log.2023-11-20.log access_log.2024-01-09.log access_log.2024-02-28.log access_log.2024-04-18.log access_log.2024-06-07.log access_log.2024-07-27.log access_log.2024-09-15.log access_log.2023-11-21.log access_log.2024-01-10.log access_log.2024-02-29.log access_log.2024-04-19.log access_log.2024-06-08.log access_log.2024-07-28.log access_log.2024-09-16.log access_log.2023-11-22.log access_log.2024-01-11.log access_log.2024-03-01.log access_log.2024-04-20.log access_log.2024-06-09.log access_log.2024-07-29.log access_log.2024-09-17.log access_log.2023-11-23.log access_log.2024-01-12.log access_log.2024-03-02.log access_log.2024-04-21.log access_log.2024-06-10.log access_log.2024-07-30.log access_log.2024-09-18.log access_log.2023-11-24.log access_log.2024-01-13.log access_log.2024-03-03.log access_log.2024-04-22.log access_log.2024-06-11.log access_log.2024-07-31.log access_log.2024-09-19.log access_log.2023-11-25.log access_log.2024-01-14.log access_log.2024-03-04.log access_log.2024-04-23.log access_log.2024-06-12.log access_log.2024-08-01.log access_log.2024-09-20.log access_log.2023-11-26.log access_log.2024-01-15.log access_log.2024-03-05.log access_log.2024-04-24.log access_log.2024-06-13.log access_log.2024-08-02.log access_log.2024-09-21.log access_log.2023-11-27.log access_log.2024-01-16.log access_log.2024-03-06.log access_log.2024-04-25.log access_log.2024-06-14.log access_log.2024-08-03.log access_log.2024-09-22.log access_log.2023-11-28.log access_log.2024-01-17.log access_log.2024-03-07.log access_log.2024-04-26.log access_log.2024-06-15.log access_log.2024-08-04.log access_log.2024-09-23.log access_log.2023-11-29.log access_log.2024-01-18.log access_log.2024-03-08.log access_log.2024-04-27.log access_log.2024-06-16.log access_log.2024-08-05.log access_log.2024-09-24.log access_log.2023-11-30.log access_log.2024-01-19.log access_log.2024-03-09.log access_log.2024-04-28.log access_log.2024-06-17.log access_log.2024-08-06.log access_log.2024-09-25.log access_log.2023-12-01.log access_log.2024-01-20.log access_log.2024-03-10.log access_log.2024-04-29.log access_log.2024-06-18.log access_log.2024-08-07.log access_log.2024-09-26.log access_log.2023-12-02.log access_log.2024-01-21.log access_log.2024-03-11.log access_log.2024-04-30.log access_log.2024-06-19.log access_log.2024-08-08.log access_log.2024-09-27.log access_log.2023-12-03.log access_log.2024-01-22.log access_log.2024-03-12.log access_log.2024-05-01.log access_log.2024-06-20.log access_log.2024-08-09.log access_log.2024-09-28.log access_log.2023-12-04.log access_log.2024-01-23.log access_log.2024-03-13.log access_log.2024-05-02.log access_log.2024-06-21.log access_log.2024-08-10.log access_log.2024-09-29.log access_log.2023-12-05.log access_log.2024-01-24.log access_log.2024-03-14.log access_log.2024-05-03.log access_log.2024-06-22.log access_log.2024-08-11.log access_log.2024-09-30.log access_log.2023-12-06.log access_log.2024-01-25.log access_log.2024-03-15.log access_log.2024-05-04.log access_log.2024-06-23.log access_log.2024-08-12.log access_log.2024-10-01.log access_log.2023-12-07.log access_log.2024-01-26.log access_log.2024-03-16.log access_log.2024-05-05.log access_log.2024-06-24.log access_log.2024-08-13.log access_log.2024-10-02.log access_log.2023-12-08.log access_log.2024-01-27.log access_log.2024-03-17.log access_log.2024-05-06.log access_log.2024-06-25.log access_log.2024-08-14.log access_log.2024-10-03.log access_log.2023-12-09.log access_log.2024-01-28.log access_log.2024-03-18.log access_log.2024-05-07.log access_log.2024-06-26.log access_log.2024-08-15.log access_log.2024-10-04.log access_log.2023-12-10.log access_log.2024-01-29.log access_log.2024-03-19.log access_log.2024-05-08.log access_log.2024-06-27.log access_log.2024-08-16.log access_log.2024-10-05.log access_log.2023-12-11.log access_log.2024-01-30.log access_log.2024-03-20.log access_log.2024-05-09.log access_log.2024-06-28.log access_log.2024-08-17.log access_log.2024-10-06.log access_log.2023-12-12.log access_log.2024-01-31.log access_log.2024-03-21.log access_log.2024-05-10.log access_log.2024-06-29.log access_log.2024-08-18.log access_log.2024-10-07.log access_log.2023-12-13.log access_log.2024-02-01.log access_log.2024-03-22.log access_log.2024-05-11.log access_log.2024-06-30.log access_log.2024-08-19.log access_log.2024-10-08.log access_log.2023-12-14.log access_log.2024-02-02.log access_log.2024-03-23.log access_log.2024-05-12.log access_log.2024-07-01.log access_log.2024-08-20.log access_log.2024-10-09.log access_log.2023-12-15.log access_log.2024-02-03.log access_log.2024-03-24.log access_log.2024-05-13.log access_log.2024-07-02.log access_log.2024-08-21.log access_log.2024-10-10.log access_log.2023-12-16.log access_log.2024-02-04.log access_log.2024-03-25.log access_log.2024-05-14.log access_log.2024-07-03.log access_log.2024-08-22.log access_log.2024-10-11.log access_log.2023-12-17.log access_log.2024-02-05.log access_log.2024-03-26.log access_log.2024-05-15.log access_log.2024-07-04.log access_log.2024-08-23.log access_log.2024-10-12.log access_log.2023-12-18.log access_log.2024-02-06.log access_log.2024-03-27.log access_log.2024-05-16.log access_log.2024-07-05.log access_log.2024-08-24.log access_log.2024-10-13.log access_log.2023-12-19.log access_log.2024-02-07.log access_log.2024-03-28.log access_log.2024-05-17.log access_log.2024-07-06.log access_log.2024-08-25.log access_log.2024-10-14.log access_log.2023-12-20.log access_log.2024-02-08.log access_log.2024-03-29.log access_log.2024-05-18.log access_log.2024-07-07.log access_log.2024-08-26.log access_log.2024-10-15.log access_log.2023-12-21.log access_log.2024-02-09.log access_log.2024-03-30.log access_log.2024-05-19.log access_log.2024-07-08.log access_log.2024-08-27.log access_log.2024-10-16.log access_log.2023-12-22.log access_log.2024-02-10.log access_log.2024-03-31.log access_log.2024-05-20.log access_log.2024-07-09.log access_log.2024-08-28.log access_log.2024-10-17.log access_log.2023-12-23.log access_log.2024-02-11.log access_log.2024-04-01.log access_log.2024-05-21.log access_log.2024-07-10.log access_log.2024-08-29.log access_log.2024-10-18.log access_log.2023-12-24.log access_log.2024-02-12.log access_log.2024-04-02.log access_log.2024-05-22.log access_log.2024-07-11.log access_log.2024-08-30.log access_log.2024-10-19.log access_log.2023-12-25.log access_log.2024-02-13.log access_log.2024-04-03.log access_log.2024-05-23.log access_log.2024-07-12.log access_log.2024-08-31.log access_log.2024-10-20.log access_log.2023-12-26.log access_log.2024-02-14.log access_log.2024-04-04.log access_log.2024-05-24.log access_log.2024-07-13.log access_log.2024-09-01.log access_log.2024-10-21.log access_log.2023-12-27.log access_log.2024-02-15.log access_log.2024-04-05.log access_log.2024-05-25.log access_log.2024-07-14.log access_log.2024-09-02.log access_log.2024-10-22.log access_log.2023-12-28.log access_log.2024-02-16.log access_log.2024-04-06.log access_log.2024-05-26.log access_log.2024-07-15.log access_log.2024-09-03.log access_log.2024-10-23.log access_log.2023-12-29.log access_log.2024-02-17.log access_log.2024-04-07.log access_log.2024-05-27.log access_log.2024-07-16.log access_log.2024-09-04.log access_log.2024-10-24.log access_log.2023-12-30.log access_log.2024-02-18.log access_log.2024-04-08.log access_log.2024-05-28.log access_log.2024-07-17.log access_log.2024-09-05.log access_log.2024-10-25.log access_log.2023-12-31.log access_log.2024-02-19.log access_log.2024-04-09.log access_log.2024-05-29.log access_log.2024-07-18.log access_log.2024-09-06.log access_log.2024-10-26.log access_log.2024-01-01.log access_log.2024-02-20.log access_log.2024-04-10.log access_log.2024-05-30.log access_log.2024-07-19.log access_log.2024-09-07.log access_log.2024-10-27.log access_log.2024-01-02.log access_log.2024-02-21.log access_log.2024-04-11.log access_log.2024-05-31.log access_log.2024-07-20.log access_log.2024-09-08.log access_log.2024-10-28.log access_log.2024-01-03.log access_log.2024-02-22.log access_log.2024-04-12.log access_log.2024-06-01.log access_log.2024-07-21.log access_log.2024-09-09.log access_log.2024-10-29.log ```
After: ``` ubuntu@ip-10-20-1-175:/var/log/avni_server/log$ ls access_log.2024-07-31.log access_log.2024-08-13.log access_log.2024-08-26.log access_log.2024-09-08.log access_log.2024-09-21.log access_log.2024-10-04.log access_log.2024-10-17.log access_log.2024-08-01.log access_log.2024-08-14.log access_log.2024-08-27.log access_log.2024-09-09.log access_log.2024-09-22.log access_log.2024-10-05.log access_log.2024-10-18.log access_log.2024-08-02.log access_log.2024-08-15.log access_log.2024-08-28.log access_log.2024-09-10.log access_log.2024-09-23.log access_log.2024-10-06.log access_log.2024-10-19.log access_log.2024-08-03.log access_log.2024-08-16.log access_log.2024-08-29.log access_log.2024-09-11.log access_log.2024-09-24.log access_log.2024-10-07.log access_log.2024-10-20.log access_log.2024-08-04.log access_log.2024-08-17.log access_log.2024-08-30.log access_log.2024-09-12.log access_log.2024-09-25.log access_log.2024-10-08.log access_log.2024-10-21.log access_log.2024-08-05.log access_log.2024-08-18.log access_log.2024-08-31.log access_log.2024-09-13.log access_log.2024-09-26.log access_log.2024-10-09.log access_log.2024-10-22.log access_log.2024-08-06.log access_log.2024-08-19.log access_log.2024-09-01.log access_log.2024-09-14.log access_log.2024-09-27.log access_log.2024-10-10.log access_log.2024-10-23.log access_log.2024-08-07.log access_log.2024-08-20.log access_log.2024-09-02.log access_log.2024-09-15.log access_log.2024-09-28.log access_log.2024-10-11.log access_log.2024-10-24.log access_log.2024-08-08.log access_log.2024-08-21.log access_log.2024-09-03.log access_log.2024-09-16.log access_log.2024-09-29.log access_log.2024-10-12.log access_log.2024-10-25.log access_log.2024-08-09.log access_log.2024-08-22.log access_log.2024-09-04.log access_log.2024-09-17.log access_log.2024-09-30.log access_log.2024-10-13.log access_log.2024-10-26.log access_log.2024-08-10.log access_log.2024-08-23.log access_log.2024-09-05.log access_log.2024-09-18.log access_log.2024-10-01.log access_log.2024-10-14.log access_log.2024-10-27.log access_log.2024-08-11.log access_log.2024-08-24.log access_log.2024-09-06.log access_log.2024-09-19.log access_log.2024-10-02.log access_log.2024-10-15.log access_log.2024-10-28.log access_log.2024-08-12.log access_log.2024-08-25.log access_log.2024-09-07.log access_log.2024-09-20.log access_log.2024-10-03.log access_log.2024-10-16.log access_log.2024-10-29.log ```
himeshr commented 3 weeks ago

Marking it as Tech type QA, to ensure we check the server state before the deploy and validate all the fixes after deployment.

Test cases

himeshr commented 1 week ago

This forwarding has been temporarily disabled in avni cloud staging, prerelease and prod. set ForwardToSyslog=no in /etc/systemd/journald.conf sudo systemctl restart systemd-journald

@1t5j0y Should we enable "ForwardToSyslog" on deployment of 10.1 to prod server.?

1t5j0y commented 1 week ago

This forwarding has been temporarily disabled in avni cloud staging, prerelease and prod. set ForwardToSyslog=no in /etc/systemd/journald.conf sudo systemctl restart systemd-journald

@1t5j0y Should we enable "ForwardToSyslog" on deployment of 10.1 to prod server.?

Yes. Updated avniproject/avni-product#1657