Instrument PCs: Log rotation automation

[GRyall]

As a systems administrator I would like there to be an automated process that moves older logs off instrument PCs to free up space. Currently this consumes support time either responding to alerts for full disks or manually running many processes between cycle.

Additionally, I would like conserver log files on Instrument PCs older than 10 days to be deleted, this will reduce effort spent freeing space on instrument PCs and match the period other logs are kept before being moved off instrument PCs.

Acceptance Criteria

• [x] #8361 has been completed
• [ ] An automated process exists which moves log files older than 10 days (except the conserver log and MySQL logs) off the instrument PCs in a similar fashion as @ChrisM-S current manually run script.
• [ ] An automated process is in place that deletes conserver logs older than 10 days
• [ ] Appropriate checks are in place that this script is running appropriately
• [ ] The process is documented in the developer wiki

Extra Information

We have other tasks already running at regular interviews, see e.g. our git processes in the blockserver.

This ticket is a result of the discussion that took place as part of #8321 .

See also: developer wiki page.

Separate ticket exists for the mysql related logs (#8364).

It is recognised that the Nagios alerts for full disks are currently used as a proxy fie the health of the controls system on the instrument PC (as an unhealthy system generates large logs). To that end this ticket depends on #8361 (Ticket on Nagios check monitoring log growth/size).

How to Test

verbose instructions for reviewer to test changes (Add before making a PR)

time in planning 01:30 23/5/24 ~00:29 20/06/24

[isaachilly]

Note that the automatic removal of logs will require modification of the IOC Log File Rate service on Nagios made in:

• https://github.com/ISISComputingGroup/IBEX/issues/8361

@FreddieAkeroyd and me (@isaachilly ) talked about what would be best. Problem: If interval the service uses as a start and end time to get difference in log dir size encompasses a time when logs are taken of this will create an inaccurate log rate.