Build mechanism to purge traffic event logs after 12 months

[plittlewood-rpt]

Background context Currently we record every request to the Platform indefinitely, even though we have processes to amalgamate the data for billing purposes. This initial data store is quickly becoming large and should be purged at an appropriate interval

Specification

• Project: Reapit.Services.HttpTrafficEvents

• 6990 logged to deal with frontend

• Build mechanism into existing traffic events service that removes data older than 12 months at the time of execution from the Platform_Billing_HttpTrafficEvents table. This should be feature flagged so the work can be done without being turned on if we get ahead of the frontend. It's likely this job will need a log somewhere of where it's processed up to so that it knows which dates to hit when removing the data
• Build single time job to remove data old than 12 months with a defined start date of the earliest date in this table
• Use the partition key 'Date' to loop through each day until 12 months ago to clear this data
• Use Marketplace SubscriptionWorker as a point of reference for background jobs

[plittlewood-rpt]

Implementation approach to be discussed in a meeting next week

[github-actions[bot]]

This issue has been updated and moved to our ‘Near Term’ column (typically completed within 1 - 4 months). We have assessed the effort required and outlined a technical specification - please take the time to review this detail. When we're ready to schedule the issue, it will be assigned to the relevant board where you can continue to track its progress to completion. For more information on our processes, please click here