Implement log housekeeping mechanism

[plittlewood-rpt]

Background context There is currently no automatic mechanism for purging old log data meaning that we keep having to either manually purge logs or increase the storage space on the OpenSearch stack. We should implement a mechanism to automatically remove old log items

Specification

• Look to see if whether a lifecycle policy in OpenSearch can be used here then we are just using the built-in mechanism (there is an old Python script which used to run to clean out old indexes but this was switched off. If we need to revisit this speak to PL who can provide access)
• OpenSearch ISM documentation looks useful here. These policies can be setup in the OpenSearch UI (Menu -> OpenSearch Plugins -> Index Management -> State Management Policies)
• Configure routine to delete indexes older than 6 months (ideally number of months should be easily changeable)

[github-actions[bot]]

This issue has been updated and moved to our ‘Mid Term’ column (typically completed within 5 - 8 months). We will assess the effort required and may outline a technical specification. When we're ready to schedule the issue, it will be moved to the ‘Near Term’ column. For more information on our processes, please click here

[HollyJoyPhillips]

PL & HP: Reviewed

[HollyJoyPhillips]

@HollyJoyPhillips to confirm date range

[github-actions[bot]]

This issue has been updated and moved to our ‘Near Term’ column (typically completed within 1 - 4 months). We have assessed the effort required and outlined a technical specification - please take the time to review this detail. When we're ready to schedule the issue, it will be assigned to the relevant board where you can continue to track its progress to completion. For more information on our processes, please click here