Managing retention of data files

MichaelHanksSF commented 5 months ago

Use AWS rules to delete files that are too old based on the age of the data within the files, rather than the date of upload to the platform. This would apply to files:

[ ] in the LA service workspace: the cleanfile folder within Current
[ ] in the LA shared files space: the cleanfile folder
[ ] in the Organisation Egress/Data Store: the cleanfile folder

This will require (we think):

[ ] Being able to add metadata to files (the year) based on pyfile setup
[ ] Being able to configure the AWS rules to use this metadata information

MichaelHanksSF commented 1 month ago

Widening this to include:

[ ] LA input files - should also have the same rules as cleanfiles and probably managed the same way

MichaelHanksSF commented 2 weeks ago

Conduct only on LA ingress folder; the existing pipeline takes care of the rest Log file for LA to be written to LA ingress

SocialFinanceDigitalLabs / sf-fons-platform

Managing retention of data files #86